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SUMMARY 


The  aim  of  this  study  was  to  gather  expert  opinion  on  the  use  of 
simulators  for  training  technicians  at  Naval  Aviation  Maintenance 
Training  Detachments  (NAMTRADETs) . The  Information  collected  was 
based  on  the  expectation  that  the  Navy  will  replace  real  equipment 
with  simulators  for  training  technicians  In  future  years. 

The  opinions  were  collected  with  three  successive  questionnaires 

completed  by  60  experts  representing  a variety  of  job  categories 

Navy  and  Marine  Instructors,  simulator  manufacturers,  university 
professors,  researchers,  and  administrators.  In  the  second  and  third 
questionnaires  of  the  series,  each  expert  was  provided  with  the  group's 
responses  to  the  preceding  questionnaire,  and  allowed  to  revise  his 
previous  opinion,  a method  of  data  collection  known  as  the  Delphi 
technique. 

The  questionnaires  asked  what  Information  an  administrator  needed 
to  decide  between  simulators  and  real  equipment  for  training  at  a 
NAMTRADET.  Also  Included  was  an  estimate  of  how  much  effort  would  be 
required  to  obtain  the  needed  Information.  Subsumed  under  Information 
requirements  were  Items  on  course  content,  economic  considerations, 
life-cycle  considerations,  repair  considerations,  physical  considerations, 
and  considerations  about  Instructors  and  students.  Additionally,  the 
experts  Judged  the  usefulness  of  various  sources  of  Information,  both 
personnel  and  documents. 

Also,  on  the  assumption  that  an  administrator  had  decided  to  procure 
a simulator,  the  questionnaires  asked  the  experts  to  judge  the  Importance 
of  addressing  various  Issues  and  to  estimate  the  frequency  with  which 
each  such  Issue  had  been  addressed  In  the  past. 

Finally,  the  experts  were  asked  to  rate  the  feasibility,  training 
effectiveness,  and  cost  effectiveness  of  simulators  for  maintenance 
training  for  nine  aircraft  systems,  separately  for  0-level  and  I-level 
maintenance. 

The  questionnaire  data  were  analyzed  separately  for  the  Navy  and 
Marine  maintenance  Instructors  (n  • 26)  and  for  all  the  other  experts 
combined  (n  > 34).  Although  both  groups  agreed  with  each  other  In  most 
areas,  a few  differences  that  may  have  significant  practical  Implications 
were  uncovered.  For  example,  the  Instructor  group  disagreed  among  them- 
selves concerning  the  feasibility,  training  effectiveness,  and  cost 
effectiveness  of  simulation  for  I-level  maintenance  training  while  the 
other  experts  were  often  In  agreement  and  generally  In  favor  of  simulation 
for  I-level  training. 

The  results  of  the  study  should  be  of  use  to  administrators,  training 
analys  j and  planners.  Instructors,  simulator  manufacturers,  cind  evaluators 
of  maintenance  simulators. 
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I.  INTRODUCTION 


In  the  last  few  years  about  30  simulators  for  aviation  main- 
tenance training  have  been  introduced  into  Naval  Aviation  Maintenance 
Training  Detachments  (NAMTRADETs) . With  the  expectation  that  many 
more  simulators  will  be  used  at  NAMTRADETs  in  future  years,  the 
Weapons  Training  Division,  Code  413,  of  Logistics/Fleet  Support  of 
the  Naval  Air  Systems  Command  tasked  the  Naval  Air  Development  Center 
(NAVAIRDEVCEN) , Code  6043,  to  report  on  the  area  of  simulators  for 
aviation  maintenance  training. 

Although  the  literature  on  simulators  in  aviation  is  vast,  almost 
all  reports  address  the  functions  of  air  crews,  not  maintenance  per- 
sonnel. Moreover,  the  studies  on  simulators  in  aviation  maintenance 
training  that  do  exist  are  mostly  descriptions  of  techniques  or  con- 
cepts still  in  the  research  and  development  stage  or  descriptions  of 
field  evaluations  of  simulators.  Most  of  what  is  known,  or  at  least 
believed,  is  grounded  not  on  experimental  data  or  theoretical  founda- 
tions but  on  experience  or  best  guesses  based  on  a general  knowledge 
of  simulation  and  training. 

Since  the  only  extensive  source  of  Information  is  expert  opinion, 
the  decision  was  made  to  systematically  collect  and  organize  such 
expert  opinion.  The  technique  selected  to  obtain  the  experts’  opinions 
was  the  Delphi  Technique,  a procedure  devised  by  the  Rand  Corporation 
about  thirty  years  ago. 

The  Delphi  Technique  has  been  used  in  a variety  of  ways  to  obtain 
expert  opinion  on  a variety  of  topics.  For  examples  of  the  uses  of 
the  Delphi  Technique,  experimental  evaluations  of  the  Technique,  as 
well  as  criticisms  of  the  Technique,  see  Cooper,  1974;  Dodge  & Clark, 
1977;  Helmer,  1966,  1967;  Linstone  & Turoff,  1975;  Quade  & Boucher, 
1968;  Sackman,  1974. 

The  Delphi  Technique,  as  employed  in  the  present  study,  will  be 
described  in  Section  II,  Data  Collection  Process. 

II.  DATA  COLLECTION  PROCESS 


a.  Delphi  Technique 

In  the  first  questionnaire  submitted  to  the  experts,  the 
Delphi  Technique  was  described  as  follows: 


The  Delphi  Technique  is  a method  for  obtaining 
and  organizing  the  independent  opinions  of  experts. 

The  technique  is  based  upon  a series  of  questionnaires. 
The  first  questionnaire  consists  of  a few  general 
questions  designed  to  evoke  discussion  of  the  selected 
topic  by  the  experts.  In  their  responses  to  the  first 
questionnaire,  the  experts  identify  the  areas  of 
relevance  and  importance,  describe  their  knowledge, 
provide  examples  and  experiences  to  support  their 
opinions,  suggest  problems,  offer  solutions.  In  short, 
in  the  first  questionnaire  the  experts  try  to  get  at 
the  essence  of  the  topic  under  consideration.  (Typically, 
the  first  questionnaire  requires  the  most  thought  and 
time  by  the  expert.) 

The  responses  to  the  first  questionnaire  are 
analyzed,  organized,  and  converted  into  individual 
items  for  the  second  questionnaire.  Thus,  by  this 
procedure,  the  experts  themselves,  not  the  coordinator, 
determine  the  content  of  the  second  (and  subsequent) 
questionnaires . 

The  format  of  the  second  questionnaire  is  like 
that  of  a typical  questionnaire:  the  expert  is 
presented  with  a series  of  items  that  he  checks  to 
indicate  that  he  agrees  or  disagrees  with  a statement, 
or  that  an  area  is  important  or  unimportant,  or  an 
approach  is  feasible  or  infeasible.  In  most  cases, 
the  response  required  is  not  simply  a Yes-No,  agree- 
disagree  response,  but  a rating  on  a scale.  For 
instance,  the  expert  might  be  asked  to  check  his 
preference  on  a 5-point  scale  varying  from  1 (strongly 
agree) to  5 (strongly  disagree). 

The  results  of  the  second  questionnaire  are 
summarized  by  the  coordinator  and  presented  to  the 
experts  in  the  third  questionnaire  for  further  consid- 
eration. The  aim  of  the  Delphi  Technique  is  to  obtain 
a consensus  of  opinion  on  the  topic  under  consideration. 

Sometimes  the  Delphi  Technique  is  continued  through 
four  or  five  or  more  questionnaires,  with  the  goal  of 
obtaining  a consensus  of  opinion  on  every  item.  Because 
of  a tight  time  schedule,  the  present  application  of  the 
Delphi  Technique  will  be  limited  to  three  successive 
questionnaires . 


i In  mnny  ways,  the  Delphi  Technique  accomplishes 

i what  one  may  accomplish  by  a well-run  conference 

! * addressing  a specific  topic.  The  Delphi  Technique, 

however,  has  certain  advantages  over  a conference: 

1.  It  gives  each  expert  the  opportunity  to  express 

! his  opinions  without  intimidation  by  other  experts 

or  by  higher  military  or  civilian  authorities. 

2.  It  allows  the  expert  to  express  his  hunches  and 

‘ half-thought-out  opinions  with  impunity  since  respon- 

; ses  are  not  attributed  to  any  expert  by  name. 

t 

3.  It  provides  the  expert  with  the  benefit  of  opinions 

f from  a variety  of  other  experts  with  different  orienta- 

tions and  biases. 

4.  It  gives  the  expert  the  chance  to  leisurely  examine 
the  opinions  of  other  experts  before  reacting. 

b.  Selection  of  Experts 

To  obtain  opinions  from  experts  who  perceived  simulators  in 
different  ways,  experts  with  a variety  of  job  categories  were  selected. 

Simulator  manufacturers,  university  professors,  administrators,  research- 
ers, and  users  of  simulators  (Navy  and  Marine  aviation  maintenance 
instructors)  participated.  From  an  initial  pool  of  about  125  experts, 
invitations  to  participate  were  extended,  by  telephone  or  in  person, 
to  62  experts.  Sixty  experts  completed  some  or  all  of  the  three 
questionnaires  (See  Appendix  A). 

The  composition  of  the  experts  is  summarized  in  Table  1. 

As  Table  1 indicates,  there  were  26  Navy  and  Marine  aviation  mainten- 
ance instructors.  This  group,  the  users  of  maintenance  simualators, 
was  purposely  large  in  order  to  allow  meaningful  comparisons  with  the 
other  experts.  In  the  text  that  follows,  the  26  instructors  will 
be  called  the  NAMTRADET  group  and  the  other  34  experts,  collectively, 
will  be  termed  the  NON-NAMTRADET  group. 

The  names  and  addresses  of  all  participants  are  listed 
in  Appendix  B. 
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Table  1 


Composition  of  experts  participating  in  Delphi  study 
of  simulators  in  aviation  maintenance  training 


Institutional  affiliation 

Category 

Navy 

Marines 

Army 

Air 

Force 

Other 

Govern- 

ment 

Industry 

Univer- 

sity 

Total 

Aviation 

maintenance 

instructor 

23 

3 

26 

Administrator 

10 

1 

1 

1 

4 

1 

18 

Researcher 

3 

4 

1 

5 

1 

14 

Consultant 

2 

2 

Total 

36 

3 

5 

2 

1 

11 

2 

60 

c.  Questionnaire  One 

Appendix  C presents  Questionnaire  One,  the  material  that 
accompanied  the  questionnaire,  and  the  cover  letter.  The  description  of 
Che  Delphi  Technique  is  omitted  from  Appendix  C because  it  was  given 
above.  The  material  accompanying  Questionnaire  One  oriented  the  expert 
to  the  Delphi  Technique,  described  the  training  situation  at  a NAMTRADET, 
defined  operational  (real)  equipment  and  simulators,  and  described  main- 
tenance levels  in  the  Navy.  The  instructions  to  Questionnaire  One 
pointed  out  that  the  study: 

(1)  is  limited  to  maintenance  training  at  NAMTRADETs^ 

(2)  includes  both  0-level  and  I-level  (but  not  depot 
level)  maintenance  training,  and 

(3)  is  limited  to  simulator  systems  that  cost  no  more 
than  $200,000. 

Part  of  Questionnaire  One  consisted  of  four  essay  questions. 
Here  is  one  of  the  four  questions: 
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(1)  General  Comparisons. 

If  you  were  serving  as  an  advisor  to  Naval 
administrators  and  were  asked  to  list  the  advan- 
tages and  disadvantages  of  simulators  and  the 
advantages  and  disadvantages  of  operational 
equipment  for  aviation  maintenance  training, 
how  would  you  respond? 

Another  part  of  Questionnaire  One,  namely.  Question  V, 
required  the  expert  to  rate,  on  5-point  scales,  the  feasibility, 
training  effectiveness,  and  cost  effectiveness  of  simulation  for 
nine  aircraft  systems  (e.g.,  propulsion,  electrical  and  instru- 
ment), separately  for  0-level  and  I-level  maintenance.  Question 
V was  included  because  at  least  two  Navy  agencies  had  issued  rec- 
ommendations for  simulating  (or  not  simulating)  aircraft  systems 
for  use  in  training,  without  documentation  for  their  recommenda- 
tions. Responses  to  Question  V should  provide  a firmer  basis  for 
such  recommendations. 

The  development  of  Questionnaire  One,  as  well  as  subse- 
quent questionnaires,  was  aided  by  inputs  from  members  of  NAVAIRDEVCEN , 
Code  604,  and  from  aviation  maintenance  technicians  assigned  to  NAS 
Willow  Grove,  Pennsylvania. 

d.  Questionnaire  Two 

Appendix  D presents  Questionnaire  Two  along  with  the  cover 
letter.  The  sections  on  the  Delphi  Technique  (given  above)  and  on 
the  background  material  (given  in  Appendix  C)  are  omitted  from  Appen- 
dix D,  although  they  were  included  with  the  material  sent  to  the  ex- 
perts. 


The  development  of  Questionnaire  Two  was  based  on  the 
responses  to  Questionnaire  One,  along  with  some  additional  material 
generated  by  the  authors.  Section  I of  Questionnaire  Two  assumed 
that  an  administrator  faced  the  problem  of  deciding  between  a simula- 
tor and  operational  equipment  to  be  used  in  a course  at  a NAMTRADET. 
In  Section  I the  expert  was  asked  to  judge  the  utility  of  given  in- 
formation, and  to  judge  the  difficulty  of  obtaining  such  information. 
Section  I also  asked  the  expert  to  evaluate  possible  sources  of  in- 
formation, both  personnel  and  documents. 

Section  II  of  Questionnaire  Two  assumed  that  the  adminis- 
trator had  decided  to  use  a simulator,  and  asked  the  expert  (a)  to 
evaluate  the  issues  to  be  considered  to  Implement  the  simulator  de- 
cision, and  (b)  to  estimate  how  frequently  these  issues  had  been 
addressed  in  the  past. 


5 


The  second  part  of  the  development  of  Questionnaire  Two 
consisted  of  summarizing  the  responses  to  Question  V of  Question- 
naire One,  namely,  the  material  on  feasibility,  training  effective- 
ness and  cost  effectiveness  of  simulators  for  0-level  and  I-level 
maintenance  training.  The  procedure  was  to  tally  the  number  of 
experts  selecting  each  of  the  five  alternatives  for  each  aircraft 
system  for  each  category  (e.g.,  feasibility),  convert  into  percent- 
ages, and  round  each  percentage  to  the  nearest  whole  number.  (See 
Appendix  A for  details  on  the  number  of  questionnaires  completed.) 

The  group's  percentages  were  inserted  in  the  tables 
which  were  then  resubmitted  to  the  experts  as  Section  III  of 
Questionnaire  Two.  For  each  item,  each  expert's  own  selection 
was  circled  in  red  to  remind  him  of  his  previous  selection  (on 
Questionnaire  One)  while  he  examined  the  group's  percentages  to 
the  same  item.  The  expert  was  allowed  to  retain  his  previous 
selection  or  change  to  another  alternative. 

e.  Questionnaire  Three 

Appendix  E presents  Questionnaire  Three,  along  with  its 
cover  letter  and  a personnel  data  sheet. 

The  development  of  Questionnaire  Three  consisted  of  merely 
summarizing  the  results  of  Questionnaire  Two  and  converting  to  per- 
centages. (See  Appendix  A on  the  details  of  the  number  of  experts 
contributing  to  the  percentage  calculations.) 

A few  new  items,  suggested  by  the  experts  when  completing 
Questionnaire  Two,  were  included  in  Questionnaire  Three. 

The  tables  on  feasibility,  cost  ef fectivenesss , and  train- 
ing effectiveness,  for  0-level  and  I-level  maintenance,  initially 
presented  in  Questionnaire  One  and  resubmitted  in  Questionnaire  Two, 
were  omitted  from  Questionnaire  Three. 

III.  FINAL  RESULTS 

This  section  presents  the  tables  of  Questionnaire  Three  and 
the  tables  on  simulators  for  0-level  and  I-level  maintenance 
training  from  Questionnaire  Two. 

a.  Groups 

* Although  the  questionnaires  were  presented  to  the  experts 
as  if  the  experts  comprised  a single  group,  in  the  analysis  given 
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here,  the  total  group  was  subdivided  into  two  sub-groups:  the 
NAMTRADET  group,  which  consisted  of  the  Navy  and  Marine  mainten- 
ance instructors,  and  the  NON-NAMTRADET  group,  which  included  all 
others.  The  NAMTRADET  group  consisted  of  26  experts;  the  NON- 
NAMTRADET  group  consisted  of  34  experts.  For  each  group  the  data 
were  analyzed  and  summarized  separately,  and  then  the  two  groups 
were  compared. 

b . Consensus 

One  aim  of  the  Delphi  Technique  is  to  determine  the  items 
on  which  the  experts  are  in  agreement,  i.e.,  the  items  on  which 
there  is  a consensus  of  opinion.  In  Appendix  F,  the  problem  of 
defining  consensus  is  discussed  and  a mathematical  index  of  con- 
sensus is  derived:  if  the  interquartile  range,  namely,  P,,  ~ ^25’ 
of  a distribution  of  judgements  to  an  item  is  less  than  1;30, 
consensus  exists;  if  the  interquartile  range  is  equal  to  or  greater 
than  1.30,  no  consensus  exists. 

c.  Data  analyses 

For  each  item  of  the  tables, the  25th,  50th  (the  median), 
and  75th  percentiles  of  the  distribution  of  judgements  were  computed. 
Also,  the  interquartile  range,  namely,  P75  - P25.  was  computed  to 
determine  if  a consensus  of  opinion  existed. 

For  each  item,  the  distribution  of  judgements  of  the 
! NAMTRADET  group  was  compared  with  the  distribution  of  judgements 

t of  the  NON-NAMTRADET  group  by  the  two-sample  Kolmogorov-Smirnof f 

I test  (See  Appendix  F) . 

d.  Reordering  of  items 

; For  each  section  of  each  table  to  be  presented,  the  items 

were  listed  according  to  two  criteria.  First,  the  items  on  which 
a consensus  of  opinion  existed  for  either  group  were  listed  first. 
Second,  within  the  list  of  consensus  items,  the  order  followed 
according  to  the  value  of  the  median  (P50) — the  lower  the  median, 
the  earlier  in  the  listing.  When  two  or  more  categories  of  respon- 
ses existed  for  an  item  (e.g.,  the  categories  "Need  for  information" 
and  "Effort  required  to  obtain  information"),  the  ordering  was  based 
on  the  first  category  listed  for  the  item.  [The  non-consensus  items 
followed  the  consensus  items,  according  to  the  value  of  the  median.] 
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The  principal  purpose  of  reordering  the  items  was  to 
present,  in  a simple  fashion,  items  on  which  the  experts  agreed, 
listed  according  to  the  importance  attributed  to  the  item. 

e.  Reading  tables 

Table  2 is  an  aid  to  reading  Tables  I,  II,  III,  IV,  and 
V that  follow.  The  format  and  symbols  illustrated  in  Table  2 were 
derived  to  summarize  the  results  of  the  study  without  getting 
bogged  down  in  numerical  trivia. 

Consensus,  as  discussed  earlier  (See  Appendix  F) , is 
defined  as  a distribution  of  judgments  that  yields  an  inter- 
quartile range  less  than  1.30;  that  is,  consensus  exists  when 
(P75  - P25)-^1.30.  In  Table  2,  an  arrow  represents  consensus. 

A dashed  arrow  represents  consensus  for  the  NAMTRADET  group;  a 
solid  arrow  represents  consensus  for  the  NON-NAMTRADET  group. 

The  scale  on  top  of  Table  2 represents  the  five  alterna- 
tives, 1,  2,  3,  4,  5 and  the  vertical  line  on  each  side  of  a scale 
number  represents  the  lower  and  upper  limits  of  that  number;  thus 
the  vertical  line  to  the  left  of  1 represents  the  value  0.50  and 
the  vertical  line  to  the  right  of  1 represents  1.499.  Similarly,  the 
vertical  line  to  the  left  of  2 designates  1.50  (or  1.499)  and  the 
vertical  line  to  the  right  of  2 designates  2.499;  etc.. 

An  arrow  points  to  the  median  (P50)  of  the  distribution 
of  responses,  referred  to  the  scale  just  discussed.  Thus,  in  item 
20,  for  the  Effort  category,  for  the  NON-NAMTRADET  group  the  arrow 
designates  P5Q  * 2.3,  which  falls  in  the  alternative  Slight  Effort. 
For  the  same  item  and  category,  the  arrow  for  the  NAMTRADET  group 
indicated  P50  = 1.4,  which  falls  in  the  alternative  Little  Effort. 

When  both  the  NAMTRADET  group  and  the  NON-NAMTRADET  group 
demonstrated  consensus,  and  their  medians  differed  by  0.25  or  less, 
the  data  for  both  groups  were  represented  by  a single  double-headed 
arrow,  as  in  item  1 of  Table  2. 

Nonconsensus  in  Table  2 is  represented  by  a horizontal  line 
and  a dot.  The  line  and  dot  refer  to  the  scale  discussed  above. 

The  left  end  of  the  horizontal  line  designates  P25*  the  right  end 
designates  ^751  and  the  dot  designates  P50*  A dashed  line  repre- 
sents the  NAMTRADET  group;  a solid  line  represents  the  NON-NAMTRADET 
group.  In  item  1 of  Table  2,  the  dashed  horizontal  line  indicates 
non-consensus  for  the  NAMTRADET  group,  with  distribution  values  of 
P25  “ 1.5,  P50  * 2.3,  and  P75  * 3.1. 
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Some  Items  in  Tables  I - V referred  both  to  simulators  and 
to  operational  (real)  equipment.  In  such  cases,  Instead  of  repeat- 
ing the  Item,  the  It^im  was  presented  once  but  the  expert  made  two 
judgments,  one  considering  the  Item  as  referring  to  a simulator  (S) 
and  the  other  considering  the  Item  as  referring  to  operational 
equipment  (0).  Item  20  of  Table  2 Illustrates  such  an  Item. 

An  asterisk  (*)  In  a box  Indicates  that  the  NAMIRADET  and 
the  NON-NAMTRADET  distributions  differed  significantly  (.05  level, 
two  tailed,  two-sample,  Kolmogorov- Smirnoff  test).  The  Effort 
category  of  Item  20  Indicates  such  a significant  difference  between 
the  distributions  of  the  two  groups. 

f.  Results 

The  final  results  of  the  Delphi  study  are  presented  In 
Tables  I through  V. 

A few  general  remarks  may  be  made  about  the  tables.  Table  I, 
concerning  the  Information  needed  to  decide  between  simulators  and 
operational  equipment,  indicates  that,  on  about  80%  of  the  items,  both 
the  NAMTRADET  group  and  the  NON-NAMTRADET  group  indicate  a consensus 
of  opinion,  the  opinion  being  that  the  need  for  information  is  crucial 
or  extremely  useful.  Both  groups  also  agree.  In  general,  that  a 
moderate  effort  would  be  required  to  obtain  such  Information. 

Table  II  indicates  a reasonable  agreement  for  both  groups  on 
what  sources,  both  personnel  and  dociments,  should  be  consulted  to 
acquire  the  needed  information. 

As  noted  earlier,  many  of  the  individual  items  in  Table  III 
are  Items  that  summarize  several  items  of  Table  I.  For  example, 

"Verify  that  course  objectives  are  met  with  the  simulation  system" 
summarizes  a number  of  items  in  the  Course  Content  section  of  Table  I. 
In  Table  III  all  experts  agree  that  the  need  to  address  most  Issues 
was  crucial  or  extremely  useful.  There  was,  however,  generally  no 
agreement,  for  either  group,  on  the  past  performance  in  addressing 
an  issue. 

Table  IV,  concerning  organizational- level  maintenance  training. 
Indicates  that  both  the  NAMTRADET  and  the  NON-NAMTRADET  groups  concur 
In  the  opinion  that  simulators  are  more  cost  effective  than  operational 
equipment.  With  regard  to  the  feasibility  of  simulation  for  different 
aircraft  systems,  however,  the  NON-NAMTRADET  experts  agree,  among 
themselves,  that  simulation  is  definitely  feasible  for  all  systems 
except  propulsion  while  the  NAMTRADET  members  agree,  among  themselves. 


Table  2 

Illustration  of  table  items  and  definitions  for  table  symbols 


Need  for 
information 


Effort  required  to 
obtain  information 


U.  oc 

UJ  o 

UJ  u. 

H-  U. 

< UJ 

“■  oc 


Item  of  information 


1.  THE  TRAINING  OBJECTIVES  OF 
THE  COURSE 


6.  LISTS  OF  FAULTS  THAT  CAN 
AND  CANNOT  BE  INSERTED 
INTO  THE  DEVICE 


20.A  DESCRIPTION  OF  THE  PHYS- 
ICAL ENVIRONMENT  ON  THE 
JOB  ((.g..  HEAT,  NOISE) 


22. THE  PHYSICAL  FIDELITY  (•.g.. 
SIZE.  WEIGHT)  REQUIRED  TO 
MEET  THE  COURSE  OBJEC- 
TIVES 


SYMBOL 


INTERPRETATION 

NAMTRAOET  GROUP,  CONSENSUS 


P25  ’’sO^EDIAN) 


NON-NAMTRADET  GROUP,  CONSENSUS 


BOTH  GROUPS,  CONSENSUS 


NAMTRAOET  GROUP,  NONCONSENSUS 


ARROW  POINTS  TO 
MEDIAN  ON  SCALE 
1,2,  3, 4,  5 ON  TOP 


i'*2S,  **50,  '*75  REFERRED 
TO  SCALE  ON  TOP 

OPERATIONAL  EQUIPMENT 

STATISTICALLY  SIGNIFICANT  DIFFERENCE  BETWEEN  THE  TWO  GROUPS 


that  simulation  Is  feasible  for  only  four  systems.  For  training, 
the  NON-NAMTRADET  group  agree  that  simulation  is  more  effective  for 
most  aircraft  systems  while  the  NAMTRADET  experts  fail  to  agree  with 
each  other  on  the  training  effectiveness  for  any  system. 

Table  V,  addressing  intezmedlate- level  maintenance,  shows 
the  greatest  differences  between  the  NAMTRADET  and  NON-NAMTRADET 
groups;  in  15  of  the  27  comparisons  in  this  table,  the  two  groups 
differed  at  a statistically  significant  level.  Within  the  NAMTRADET 
group,  a lack  of  consensus  occurred  on  all  but  five  occasions,  and 
these  five  instances  of  consensus  were  votes  against  simulation. 

The  NON-NAMTRADET  experts  agreed  among  themselves  on  12  of  the  27 
occurrences,  and  all  12  agreements  were  in  favor  of  simulation. 

In  general,  the  NAMTRADET  instructors  hold  a rather  dim  view  of 
simulation  for  intermediate-level  maintenance  training  while  the 
NON-NAMTRADET  experts  feel,  at  least  for  some  systems,  that  simula- 
tion is  both  feasible  and  coat  effective. 

g.  Comments  of  experts 

The  experts  were  encouraged  to  provide  comments  when 
completing  the  questionnaires.  Such  comments,  roughly  classified 
In  categories,  are  reproduced  in  Appendix  G. 

IV.  RECOMMENDED  USE  OF  RESULTS 

The  findings  summarized  in  Tables  I - V can  serve  as  a check- 
list for  a variety  of  personnel  associated  with  simulators  for  avia- 
tion maintenance  training.  The  administrator  can  employ  the  d?ta 
tables  to  decide  whether  to  buy  a simulator  or  operational  equipment, 
and  the  areas  he  should  consider  if  he  decides  to  buy  a simulator. 

Others  concerned  with  task  analyses,  learning  objectives,  design, 
manufacture,  support  and  updating,  and  other  areas  can  consult  the 
tables  as  a starting  point  for  their  work  assignments.  The  data  of 
Tables  I - V should  be  of  use  to  administrators,  training  analysts 
and  planners.  Navy  and  Marine  Instructors,  manufacturers,  and  evalua- 
tors of  maintenance  simulators. 

Table  V indicates  that  the  NAMTRADET  instructors  disagree  among 
themselves  on  the  feasibility,  training  effectiveness,  and  cost 
effectiveness  of  simulators  for  I-level  maintenance  training.  This 
disagreement  may  result  from  a lack  of  experience,  since  most  main- 
tenance slaalators  in  the  fleet  are  0- level,  not  I-level  simulators. 

When  the  NAMTRADET  instructors  do  agree  with  each  other,  they  have  a 
negative  attitude  toward  I-level  maintenance  simulators.  This  finding 
suggests  problems  of  user  acceptance  will  probably  develop  when  I-level 
maintenance  simulators  are  introduced  into  the  fleet  in  quantity.  Steps 
to  facilitate  user  acceptance  should  be  instituted  now. 
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Table  I.  Information  Required  for  Decision  Making 
COURSE  CONTENT  CONSIDERATIONS 


COURSE  CONTENT  CONSIDERATIONS 


Need  for 
information 


Effort  required  to 
obtain  information 


oc  X 

U UJ 


2 5 


Item  of  information 


6.  LISTS  OF  FAULTS  THAT  CAN 
AND  CANNOT  BE  INSERTED 
INTO  THE  DEVICE 


7.  THE  CRITERIA  USED  TO 
EVALUATE  THE  STUDENT'S 
PERFORMANCE  FOR  EACH 
TASK 


8.  THE  TOOLS,  TEST  EQUIPMENT, 
AND  DOCUMENTATION  AVAIL- 
ABLE TO  THE  STUDENT  DUR- 
ING  TRAINING  AND  TESTING 


9.  THE  TOOLS,  TEST  EQUIPMENT, 
AND  DOCUMENTATION  AVAIL- 
ABLE TO  THE  STUDENT  TO  PER- 
FORM EACH  TASK  ON  THE  JOB 


10. THE  CAPABILITY  OF  THE  DE- 
VICE FOR  DEMONSTRATING 
SAFETY  PROCEDURES 


11.ANALYSISOF  FREQUENT  FAIL- 
URES AND  REPAIRS  OF  THE 
WEAPON  SYSTEM 


12.THE  AMOUNT  OF  FEEDBACK 
TO  THE  STUDENT  ABOUT  HIS 
PERFORMANCE 


13.THE  AMOUNT  OF  FEEDBACK 
TO  THE  INSTRUCTOR  ABOUT 
STUDENT  PERFORMANCE 


14. THE  AVAILABILITY  OF  OTHER 
COURSES  WHICH  MIGHT  MEET 
THE  OBJECTIVES  OP  THE 
PLANNED  COURSE 
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COURSE  CONTENT  CONSIDERATIONS 


Item  of  information 


IS. THE  AVAILIBILITY  OF  IN- 
STRUCTIONAL AIDS  U.a., 
LESSON  PLANS,  MOVIES) 


16.THE  AMOUNT  AND  TYPE 
ADDITIONAL  TRAINING  RE- 
QUIRED ON  THE  JOB 


OF 


17.THE  TIME  IT  SHOULD  TAKE 
THE  STUDENT  TO  PERFORM 
EACH  TASK 


IS. THE  NEED  FOR  ALTERNATIVE 
ROUTES  (i.«.,  BRANCHING)  FOR 
LEARNING 


19.THE  AMOUNT  OF  SUPERVISION 
PROVIDED  ON  THE  JOB 


20.A  DESCRIPTION  OF  THE  PHYS- 
ICAL ENVIRONMENT  ON  THE 
JOB  ((.g.,  HEAT.  NOISE) 


21. THE  NEED  TO  DEMONSTRATE 
MULTIPLE  FAULTS  FOR  A 
SINGLE  FAILURE 


22.THE  PHYSICAL  FIDELITY  (•.9.. 
SIZE,  WEIGHT)  REQUIRED  TO 
MEET  THE  COURSE  OBJEC- 
TIVES 


23.THE  NEED  TO  INCLUDE  IN- 
FREQUENTLY ENCOUNTERED 
MAINTENANCE  TASKS 


24.THE  NEED  FOR  SELF-PACED 
LEARNING 


Need  for 
information 


u 

o 

oc 

o 


UJ 

CO 

3 


UJ 

S 

UJ 

cc 

H 

X 

UJ 


ui 

CO 

3 


< 

Z 

o 

H 

O. 

O 


< 

> 

E 

H 


■ 


1 


11 


• : 
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i 


Effort  required  to 
obtain  information 


H 

cc 

O 

u. 

u. 

UJ 

UJ 


H 

CC 

O 

u. 

u. 

UJ 

1- 

Z 

2 

□ 

CO 


H 

CC 

o 

u. 

u. 

UI 

UJ 

H 

< 

CC 

UJ 

o 

o 

S 


H 

CC 

O 

u. 

u. 

UJ 

cc 

O 


H 

CC 

O 

u. 

u. 

UI 

UJ 

S 

UJ 

cc 

H* 

X 

UJ 


1 


i 


11 


i 

■ 


t 


u 


* 


n 


4 

I 


1 
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ECONOMIC  CONSIDERATIONS 


26. THE  FUNDS  AVAILABLE  FOR 
THE  PROJECT 


27.THE  COST  FOR  SPARE  PARTS 
FOR  EACH  DEVICE 


28.THE  COST  TO  MODIFY 

NAMTRAOET  FACILITIES  TO 
USE  DEVICE 


29.THE  COST  AND  NUMBER  OP  IN- 
STRUCTORS AND  SUPPORT 
PERSONNEL  WITH  EACH 
DEVICE 


s 


30.THE  COST  TO  REPAIR  EACH 
DEVICE  IF  IT  MALFUNCTIONS 


w 


ECONOMIC  CONSIDERATIONS 


Item  of  information 


Need  for 
information 


u 

z> 

cc 

o 


3 

u. 

lU 

U) 

3 


ai 

S 

lU 

cc 

H 

X 

lij 


3 

u. 

UJ 

CO 

3 


< 

Z 

o 

p 

a. 

O 


< 

> 

E 

H 


Effort  required  to 
obtain  information 


cc 

O 

u. 

u. 

Ui 

UJ 

-J 

H 

H 


cc 

0 

u. 

u. 

Ui 

H 

1 

g 

□ 

Ui 


H 

CC 

O 

u. 

u. 

Ui 

UJ 

h- 

< 

CC 

UJ 

o 

o 

S 


H 

CC 

o 

u. 

u. 

UJ 

CC 

o 


31. THE  COST  FOR  THE  ROUTINE 
MAINTENANCE  OF  EACH 
DEVICE 


i 


1 


i 


1 


32.THE  COST  TO  UPDATE  EACH 
DEVICE  AS  CHANGES  OCCUR 
IN  THE  WEAPON  SYSTEMS 


i 


1 


t 


33. THE  COST  TO  TRANSPORT  AND 
INSTALL  EACH  DEVICE 


i 

■ 


i 


i 


i 


34.POWER  AND  RELATED  COSTS 
TO  OPERATE  EACH  DEVICE 


ii 


dt 


il 


* 
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EXTREME  EFFORT 


LIFE  CYCLE  CONSIDERATIONS 


Need  for 
information 


Effort  required  to 
obtain  information 


Item  of  information 


35.THE  LENGTH  OF  TIME 
QUIRED  TO  PROCURE.  DE- 
LIVER, AND  INSTALL  THE 
DEVICE 


36. THE  AVAILABILITY  OF  INFOR- 
MATION NEEDED  TD  DEVELOP 
A TRAINING  COURSE 


37.THE  TIME  AVAILABLE  TO 
TRAIN  THE  PERSONNEL 
REQUIRED  TO  SUPPORT  THE 
IT 


38.  FOR  A NEW  WEAPON  SYSTEM, 
THE  DELIVERY  DATES  FOR 
FLEET  INTRODUCTION 


39.THE  EXPECTED  LIFE  OF  THE 
WEAPON  SYSTEM 


40.THE  NUMBER  OF  MAINTEN- 
ANCE PERSONNEL  REQUIRED 
TO  SUPPORT  THE  WEAPON 
SYSTEM 


41. THE  ANTICIPATED  FRE- 
QUENCY AND  COMPLEXITY  OF 
MODIFICATIONS  OF  THE 


ivi 


42.POLITICAL  PREFERENCES  FOR 
DIFFERENT  DEVICES 


43.THE  ANTICIPATED  CAPABILI- 
TIES OF  SIMULATORS  IN  THE 
NEXT  FEW  YEARS 


EXTREME  EFFORT 


REPAIR  CONSIDERATIONS 


Need  for 
information 


Effort  required  to 
obtain  information 


Item  of  information 


.REQUIREMENTS  FOR  SPECIAL 
TOOLS  AND  TEST  EQUIPMENT 
TO  REPAIR  EACH  DEVICE 


45.  REQUIREMENTS  FOR  SPECIAL- 
LY TRAINED  PEOPLE  TO  RE- 
PAIR EACH  DEVICE 


46.AVAILABILITY  DF  SPARE 

PARTS  IF  DEVICE  REQUIRES 
REPAIR 


47.ESTIMATES  OF  FREQUENCY  OF 
REPAIR  FOR  EACH  DEVICE 


48.  EXTENT  OF  FACTORY  SUP- 
PORT REQUIRED  TO  REPAIR 
DEVICE 


S2.REQUIREMENTS  FOR  SPECIAL 
DOCUMENTATION  TO  REPAIR 
EACH  DEVICE 


PHYSICAL  CONSIDERATIONS 


Need  for 
information 


Effort  required  to 
obtain  information 


Item  of  information 


53.THE  SCHEDULING  FLEXIBIL- 
ITY OF  THE  COURSE  (i.e.,  WHAT 
WILL  HAPPEN  IF  THE  DEVICE 
IS  INOPERATIVE?) 


54.THE  NUMBER  OF  STUDENTS 
WHO  COULD  USE  EACH  DEVICE 
AT  THE  SAME  TIME 


SS.THE  NUMBER  OF  STUDENTS  TO 
BE  TRAINED  WITH  THE  DEVICE 
IN  ONE  YEAR 


S6.THE  DURABILITY  OF  EACH 
DEVICE  UNDER  FREQUENT 
USE 


S7.THE  ELECTRICAL  OR  OTHER 
POWER  REQUIREMENTS  FOR 
EACH  DEVICE 


EXTREME  EFFORT 


PHYSICAL  CONSIDERATIONS 


Need  for 
information 


Effort  required  to 
obtain  information 


O o 

u.  u- 

u. 

u UJ 


OC 

H 2 

cc  u. 


CONSIDERATIONS  ABOUT  INSTRUCTORS 
AND  STUDENTS 


Need  for 
information 


Effort  required  to 
obtain  information 


Item  of  information 


64.THE  LIKELIHOOD  THAT  THE 
DEVICE  MAY  CAUSE  INJURY 
TO  THE  STUDENT 


65.THE  AMOUNT  OF  TRAINING  OF 
INSTRUCTORS  REQUIRED  TO 
USE  EACH  DEVICE 


66.THE  EXTENT  OF  INSTRUC- 
TOR'S EXPERIENCE  WORKING 
WITH  MAINTENANCE  OF  THE 
SYSTEM  


67. THE  EXTENT  OF  THE  INSTRUC- 
TOR'S EXPERIENCE  TEACHING 
THE  COURSE 


68. THE  ATTITUDES  OF  STUDENTS 
ABOUT  EACH  DEVICE 


S9.THE  EDUCATIONAL  BACK- 
GROUND OF  POTENTIAL 
STUDENTS 


70. THE  ATTITUDES  OF  INSTRUC- 
TORS ABOUT  EACH  DEVICE 


H 

OC 

O 

U-  i- 

U.  cc 

UJ  o 
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EXTREME  EFFORT 


CONSIDERATIONS  ABOUT  INSTRUCTORS 
AND  STUDENTS 


Table  II.  Sources  of  Information  for  Decision  Making 
PERSONNEL 


Need  to  consult 


8 

21  - 

I I 


Source  of  information 


2 3 


1.  INSTRUCTORS  OF  THE  COURSE 


2.  MANUFACTURERS  OF  SIMULATORS 


3.  RESEARCH  PERSONNEL  EXPERIENCED  WITH  SIMULATORS  USED  FOR 
TRAINING 


4.  TRAINING  ANALYSTS 


S.  PERSONNEL  DOING  THE  MAINTENANCE  JOB  IN  THE  FLEET 


6.  COMPANIES  WITH  THE  POTENTIAL  FOR  MANUFACTURING  SIMULATORS 


7.  INSTRUCTORS  WHO  ARE  TEACHING  WITH  SIMULATORS 
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PERSONNEL 


Source  of  information 


8.  NAVY  MANAGEMENT 


9.  INSTRUCTORS  TEACHING  SIMILIAR  COURSES 


10.NAVY  SUPPLY  SPECIALISTS 


11. NAVY  PROCUREMENT  SPECIALISTS 


12.STUOENTS  OF  THE  COURSE 


13. NAVY  SCHOOL  ADMINISTRATORS 


14.NAVY  SAFETY  SPECIALISTS 


1S.AIRCRAFT  MANUFACTURER 


18.GRADUATES  OF  THE  COURSE 


Need  to 
consult 


> 

cc 

o 

I- 

< 

o 

z 

< 

s 


ui 

CO 

o 

> 

oc 

UJ 

> 

> 

ffi 

< 

CQ 

o 

oc 

0. 


=> 

u. 

UJ 

(O 

3 


< 

Z 

o 

H 

a. 

O 


> 

OC 

< 

UJ 

u 

UJ 

z 

z 

3 


11 


> k 


± 


17.MANUFACTURERS  OF  AIRCRAFT  COMPONENTS 


DOCUMENTS 


Source  of  information 


18.MAINTENANCE  MANUALS  FOR  THE  WEAPON  SYSTEM 


19. MAINTENANCE  PLANd.t.,  WRITTEN  RECORD  OF  MAINTENANCE 
CONCEPT  FOR  SYSTEM  AND  EQUIPMENT,  INCLUDING  MAIN- 
TENANCE TASKS,  LEVEL  OF  REPAIR  ANALYSIS,  AND  SUPPORT 
AND  EQUIPMENT) 


20.PERSONNEL  AND  TRAINING  PLAN  li.•..  WRITTEN  RECORD  OF  PERSONNEL 
AND  TRAINING  REQUIRED  TO  OPERATE  AND  SUPPORT  THE  EQUIPMENT) 


21.TECHNICAL  DATA  PLAN  WRITTEN  DESCRIPTION  OF  ALL  INFOR- 
MATION AIDS  NECESSARY  TO  OPERATE  AND  SUPPORT  THE  EQUIPMENT) 


22. RESEARCH  AND  EVALUATION  REPORTS  ON  MAINTENANCE  SIMULATORS 


23.  FLIGHT  MANUALS  OF  THE  WEAPON  SYSTEM 


24.  FACILITIES  REQUIREMENTS  PLAN  (i.t.,  WRITTEN  RECOROOF  SHIP- 
BOARD, SHORE,  OPERATIONAL,  MAINTENANCE  AND  TRAINING 
FACILITIES) 

25. TRANSPORTATION  AND  HANDLING  PLANIi.*.,  WRITTEN  REPORT  OF 
PACKAGING,  HANDLING,  STORAGE,  AND  TRANSPORTATION 
REQUIREMENTS  FOR  EQUIPMENT) 


26.  RELEVANT  GOVERNMENT  INSTRUCTIONS 
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UNNECESSARY 
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Table  III.  Issues  to  be  Addressed  in  Procuring  Simulators 


Need  to  address 
an  issue 


Past  performance  in 
addressing  an  issue 
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Issue  to  be  addressed 


1.  VERIFY  THAT  COURSE  OB- 
JECTIVES ARE  MET  WITH  THE 
SIMULATOR  SYSTEM 
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FECTIVENESS  OF  THE  SIMU- 
LATOR SYSTEM  BY  TESTING 
STUDENT  PERFORMANCE 


4.  ESTABLISH  PROGRAMS  TO 
TEACH  INSTRUCTORS  HOW  TO 
USE  SIMULATORS  FOR  TRAIN- 
ING   


B.  INSURE  THE  TRAINING  SITE  IS 
ADEQUATELY  PREPARED  FOR 
INSTALLATION  OF  SIMULATOR 


6.  INSURE  ADEQUACY  OF  IN- 
STRUCTOR TRAINING  FOR  USE 
OF  SIMULATOR  SYSTEM 


Table  III 


Need  to  address 


Past  performance  in 
addressing  an  issue 
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Issue  to  be  addressed 


7.  INSURE  PROVISIONS  POR 
UPOATES  ANO  CHANGES 


8.  VERIFY  ALL  COSTS  ASSOCI- 
ATED WITH  THE  SIMULATOR  ARE 
WITHIN  ESTIMATES 


TIME  SCHEDULES  FOR  DEVEL- 
OPMENT AND  INSTALLATION 
OF  THE  SIMULATOR 


10.ESTABLISH  EFFECTIVE 

CHANNELS  OF  COMMUNICA- 
TION BETWEEN  INSTRUCTORS 
ANO  MANAGEMENT 


11.VERIFY  THAT  ACTUAL  REPAIR 
FREQUENCY  ANO  "DOWNTIME" 
ARE  WITHIN  ESTIMATES 


12.INSURE  ADEQUATE  HUMAN 
FACTORS  DESIGN  REQUIRE- 
MENTS HAVE  BEEN  MET 


13. EVALUATE  INSTRUCTOR  PER- 
FORMANCE 


14.ASSESS  ALL  EFFECTS  RESULT- 
ING FROM  INTRODUCTION  OF 
SIMULATOR  SYSTEM 


IS.  ESTABLISH  ANO  MONITOR 
TIME  SCHEDULES  FOR 
NAMTRAOET  COURSE 
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Tabta  IV.  Ftasibility,  Trainins  Effactivenass,  and  Cost  Effactivanass  of  Simulators 
for  Organizational- Laval  Aircraft  Maintananca  Training  at  NAMTRADETS 


Faasibility 


T raining 
Effactivanass 


Cost 

Effactivanass 


Tha  simulation  of  most 
or  all  of  this  system  is: 


Compared  to  operational 
equipment,  simulators  are: 
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Aircraft  System 


1.  ELECTRtCA>.  a INSTRUMENT 


Table  V.  Feasibility,  Training  Effectiveness,  and  Cost  Effectiveness  of  Simulators 
for  Intermediate-Level  Aircraft  Maintenance  Training  at  NAMTRAOETS 


Feasibility 

Training 

Effectiveness 

Cost 

Effectiveness 

The  simulation  of  most  or 
all  of  this  system  is: 

Compared  t 
equipment,  s 

0 operational 
imulators  are: 
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Aircraft  System 


1.  ELECTRICAL  & INSTRUMENT 


2.  COMMUNICATIONS  AND 
NAVIGATION 


3.  ANTISUBMARINE  WARFARE 


4.  AUTOMATIC  FLIGHT  CONTROLS 


$.  RECONNAISSANCE 


E.  ARMAMENT/WEAPONS 
DELIVERY 


7.  ENVIRONMENTAL/EGRESS 


t.  STRUCTURE,  HYDRAULICS, 
a FLIGHT  CONTROLS 


S.  PROPULSION 
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Note. — Each  questionnaire  was  mailed  to  all  62  experts. 

As  discussed  in  the  text,  the  responses  of  the  experts 
to  one  questionnaire  were  summarized  in  percentages  and 
resubmitted  in  a subsequent  questionnaire.  The  percentages 
were  computed  on  the  final  day  specified  for  the  return  of  a 
questionnaire.  For  example,  50  experts  returned  Question- 
naire One,  but  two  chose  not  to  complete  the  0-  and  I-level 
tables  and  two  others  returned  the  questionnaire  too  late  for 
inclusion  in  the  computation  of  the  percentages.  Thus,  the 
percentages  given  in  the  0-  and  I-level  tables  of  Question- 
naire Two  are  based  on  46  experts.  The  preceding  sentence 
must  be  qualified  because  not  all  of  the  46  experts  completed 
every  one  of  the  54  selections  for  the  0-  and  I-level  tables. 
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Questionnaires  returned 


Numbers  of  experts 


All  three  questionnaires 

Only  Questionnaires  One  and  Two 

Only  Questionnaires  One  and  Three 

Only  Questionnaires  Two  and  Three 

Only  Questionnaire  One 

Only  Questionnaire  Two 

Only  Questionnaire  Three 

None 


APPENDIX  A 


Participation  of  experts  in  the  Delphi  study 


A total  of  62  experts  were  invited  to  participate  in  the 
Delphi  study.  The  numbers  of  experts  returning  the  question- 
naires are  summarized  in  Table  A-1. 


Table  A-1 


Number  of  experts  returning  questionnaires  of  Delphi  study 


In  all  computations,  the  percentages  were  based  on  the  number 
of  experts  completing  an  item. 


Table  A-2 

Number  of  experts  used  as  a basis  for  computing 
percentages  for  questionnaire  tables 


Table  Designation 

Number  of 

experts 

Total 

group 

NAMTRADET 

group 

NON-NAMTRADET 

group 

0-  and  I-level  tables  for 
Questionnaire  Two 

46 

— 

— 

Tables  for  Questionnaire 
Three 

52 

Tables  in  final  results 

60 

26 

34 

The  same  reasoning  and  rules  were  applied  in  the  compu- 
tation of  the  percentages  for  all  tables,  including  those 
representing  the  final  results.  In  addition,  in  computing 
the  percentages  for  the  final  results  (not  given  in  this 
report) , the  most  recent  input  from  an  expert  was  counted 
towards  the  final  results.  Thus,  if  an  expert  retum-u  only 
Questionnaires  One  and  Two,  his  data  of  Questionnaire  Two 
were  included  in  the  final  results. 


i 
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APPENDIX  B 


Expert  participants  In  Delphi  study  of  simulators 
In  aviation  maintenance  training 

A.  NAMTRADET  Instructors 


ADCS  Kenneth  I.  Bryant 

NAMTD  1048 

NAS  Pensacola,  FL  32508 

ATI  Robert  L.  LaPorte 

NAMTD  1026 

NAS  Norfolk,  VA  23511 

1 

1 

1 

i 

ATI  David  L.  Cooper 

NAMTD  1048 

NAS  Pensacola,  FL  32508 

ATCS  Peter  McConnell 

NAMTD  1008 

NAS  Miramar,  CA  92145 

AEl  Donald  R.  Cornelius 

NAMTD  1069 

NAS  North  Island 

San  Diego,  CA  92135 

AEC  Gary  McCullough 

NAMTD  1011 

NAS  Jacksonville,  FL  32212 

1 

j 

GYSGT  Don  H.  Foster 

NAMTD  1078 

MCAS  El  Toro 

Santa  Ana,  CA  92709 

GYSGT  James  R.  McNulty 

NAMTD  1006 

MCAS  Cherry  Pt.,  NC  28533 

AFCM  James  W.  Frush 

NAKTD  1002 

NAS  Key  West,  FL  33040 

AT2  Stuart  F.  Morgan 

NAMTD  1014 

NAS  Oceana  Bldg.  240 

Virginia  Beach,  VA  23460 

AMSC  Thomas  R.  Griffith 

NAMTD  1011 

NAS  Jacksonville,  FL  32212 

ATC  James  E.  Morrison 

Box  10  NAMTD  1018 

NAS  Meridian,  MS  39301 

ATC  Garry  K.  Hanson 

Box  10  NAMTD  1018 

NAS  Meridian,  MS  39301 

AQl  James  A.  Nuessle 

NAMTD  1001 

NAS  Whldbey  Island 

Oak  Harbor,  WA  98277 

AMHl  J.  Howlngton 

NAMTD  1069 

NAS  North  Island 

San  Diego,  CA  92135 

AECS  Curtis  G.  Olson 

C.  0.  AIRANTISUBRON  33 
% FPO  San  Francisco,  CA  96601 

AEC  James  J.  Jozwlak 

NAMTD  1018 

NAS  Meridian,  MS  39301 

MSGT  K.  D.  Osborn 

NAxMTD  1078 

MCAS  El  Toro 

Santa  Ana,  CA  92709 

[ 
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AQC  J.  M.  Owens 
NA^^^D  1008 

NAS  Miramar,  CA  92145 


AVCM  Ronald  E.  Rumpf 
NAMTD  1026 
Norfolk,  VA  23505 


AMCS  L.  Purviance 
NAMTD  1034 

NAS  Cecil  Field,  FL  32215 

AMHC  Salvatore  Rao 
NAMTD  1018 

NAS  Meridian,  MS  39301 

AVCM  Jerry  D.  Rohrer 
NAMTD  1002 

NAS  Key  West,  FL  33040 


NON-NAMTRADET  experts 

Mr.  D.  B.  Adams 
NAVAIRSYSCOM  AIR  4131 
Washington,  DC  20360 

Robert  J.  Biersner,  Ph.D. 
LCDR  MSC  USN 
Naval  Submarine  Medical 
Research  Lab. 

Naval  Submarine  Base 
New  London 
Box  900 

Groton,  CT  06340 

Dr.  Harold  Booher 
Code  308 

Naval  Personnel  Research 
& Development  Center 
San  Diego,  CA  92152 

LCDR  Ray  S.  Bonder 
General  Programs  Officer 
NAMTRAGRU  34 
NAS  Memphis 
Millington,  TN  38054 


ATCS  James  W.  Scrlvner 

VAW-124  Avionics 

FPO  New  York,  NY  09501 

ATC  Walter  J.  Todd 
NAMTD  1034 

NAS  Cecil  Field,  FL  32215 

AQC  John  F.  Williams,  Jr. 
NAMTD  1001 
NAS  Whidbey  Island 
Oak  Harbor,  WA  98277 


Dr.  Richard  Braby 

Training  Analysis  & Evaluation  Group 
Orlando,  FL  32813 

Dr.  William  H.  Crooks 
Perceptronics,  Inc. 

6271  Variel  Ave. 

Woodland  Hills,  CA  91364 


Dr.  John  P.  Foley,  Jr. 
AFHRL 

Brooks  AFB,  TX  78235 


Mr.  Ra)nnoiid  B.  Fox 
IBM  Federal  Systems  Division 
Bldg.  400/043 
9500  Godwin  Drive 
Manassas,  VA  22110 


Dr.  F.  C.  Frick 

MIT  Lincoln  Laboratory 

Lexington,  MA  02173 


Mr.  K.  C.  Hageman 
Hageman  Consulting  Services 
P.O.  Box  11409 
Ft.  Worth,  TX  76109 


Dr.  George  Lukas,  Senior  Mathematician 
Bolt,  Beranek  and  Newman,  Inc. 

50  Moulton  St. 

Cambridge,  MA  02138 

Mr.  Arthur  Mann 

San  Diego  Aeronautical  Audit  Complex 
NAS  North  Island  Bldg.  252 
San  Diego,  CA  92135 


Dr.  Donald  F.  Haggard 
U.  S.  Army  Research  Institute 
Field  Unit 
Bldg.  2423 
Fort  Knox,  KY  40121 

Mr.  Alfred  J.  Homann 
OMNIDATA,  Inc. 

7300  Route  130 
Pennsauken,  NJ  08110 


Mr.  Arthur  Marcus,  Senior  Research 
Scientist 

U.  S.  Army  Research  Institute 
1300  Wilson  Boulevard 
Rosslyn,  VA  22209 

Dr.  Lee  A.  Miller 
Honeywell , Inc . 

2600  Ridgway  Parkway 
Minneapolis,  MN  55413 


Dr.  Richard  Hurlock 
Code  304,  Development  of 
Training  Technology 
Naval  Personnel  Research 
& Development  Center 
San  Diego,  CA  92152 


Dr.  John  A.  Modrick,  Staff  Scientist 
Honeywell,  Inc. 

2600  Ridgway  Parkway 
Minneapolis,  MN  55413 


Mr.  Reid  P.  Joyce,  Principal 
Scientist 

Applied  Science  Associates, 
Inc. 

Box  158 

Valencia,  PA  16059 

Dr.  William  King 
Code  N-215 

Naval  Training  Equipment 
Center 

Orlando,  FL  32813 


CDR  R.  L.  Mudgett 
Code  315 
NAMTRAPAC 
NAS  North  Island 
San  Diego,  CA  92135 


Mr.  William  G.  Muller 
Naval  Air  Technical  Services 
Facility  (123) 

700  Robbins  Ave. 
Philadelphia,  PA  19111 


Mr.  Manuel  Lopez 
NAVAIRSYSCOM  AIR  4132 
Washington,  DC  20360 


Dr.  Marshall  Narva 

U.  S.  Army  Research  Institute 

1300  Wilson  Blvd . 

Arlington,  VA  22209 
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Dr.  Leon  H.  Nawrockl 

U.  S.  Army  Research  Institute 

1300  Wilson  Blvd. 

Arlington,  VA  22209 


AVCM  J.  R.  Nelson 
NAMTRAGRU 
NAS  Memphis 
Millington,  TN  38054 


Mr.  Robert  D.  Plunkett 
Directorate  of  Training 
Development 
USA  Signal  School 
Fort  Gordon,  GA  30905 

Mr.  Jack  Richardson,  C.P.L. 
Supervisory  General  Engineer 
Naval  Air  Systems  Command 
AIR  4134 

Washington,  DC  20361 

Dr.  J.  W.  Rlgney 
Behavioral  Technology  Lab. 

Unlv.  of  Southern  Calif. 

University  Park 

Los  Angeles,  CA  90007 

Dr.  Marty  R.  Rockway 
Technical  Director 
Technical  Training  Division 
(AFHRL/TT) 

USAF  Human  Resource  Laboratory 
Lowry  AFB,  CO  80230 

Dr.  Edgar  L.  Shrlver,  President 
Klnton,  Inc. 

100  Prince  St. 

Alexandria,  VA  22314 


Mr.  Nicholas  A.  Siecko,  Vice 
President,  Education  Research 
and  Development 

Educational  Computer  Corporation 
175  Strafford  Avenue 
Strafford,  PA  19087 

Dr.  Leonard  C.  Silvern,  Education 
Consultant 

Aeronutronic  Ford  Corp., 

The  Ford  Motor  Company 
Box  49899 

Los  Angeles,  CA  90049 

Dr.  Robert  G.  Smith,  Jr. 

Assistant  for  Personnel  Logistics 
Plans 

OPNAV  OP-987  PIO 
Washington,  DC  20350 
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The  description  of  the  Delphi  Technique,  which  was 
included  with  Questionnaire  One,  is  omitted.  It  is 
reproduced  in  the  section  on  the  Data  Collection  Process. 
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Dear 


Thank  you  again  for  agreeing  to  serve  as  an  expert  in  our  study  on 
aviation  maintenance  training  simulators. 

The  enclosed  material  provides  background  information  relevant  to 
the  studyl 

1.  The  Delphi  Technique,  the  method  to  be  used  in  this  study, 
is  described. 

2.  Definitions  and  descriptions  are  given  for: 

a.  The  Navy  training  situation. 

b.  Operational  equipment  and  simulators. 

c.  Maintenance  programs  in  the  Navy. 

Following  this  background  information  is  Questionnaire  One,  which  you 
are  asked  to  complete  and  return.  A postage-free,  addressed  envelope 
is  enclosed. 

Although  the  study  is  specific  to  maintenance  training  in  the  Navy,  it 
is  likely  that  the  results  will  be  generalizable,  at  least  to  some 
extent,  to  other  training  situations.  Thus,  even  if  you  are  not  assoc- 
iated with  the  Navy,  you  and  your  organization  should  profit  from  the 
study.  A copy  of  the  final  report  of  the  study  will  be  sent  to  you. 

I am  enclosing  my  card.  Contact  me  if  necessary.  If  I am  not  avail- 
able, ask  for  Joann  Wright  who  is  also  working  on  this  project. 

One  last  request.  The  time  schedule  on  this  project  is  very  tight. 

The  Delphi  Technique  ordinarily  takes  a lot  of  time  because  the  experts 
delay  returning  the  questionnaires.  Please  return  the  enclosed  question- 
naire promptly,  within  a couple  of  days  if  at  all  possible.  (Since  you 
are  going  to  spend  time  on  it  eventually,  why  not  do  it  now?)  If  it's 
impossible  to  get  to  it  immediately,  please  be  sure  to  return  it  within 
ten  days. 

Sincerly, 


ROBERT  M.  HERRICK 

The  opinions  expressed  or  implied  in  the  enclosed  material  : .-e  mine, 
not  the  Navy's.  No  remuneration  will  be  provided  to  the  part  . nts 
of  this  study. 
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BACKGROUND  MATERIAL 


Training  Situation 

Tha  training  imder  consideration  occurs  at  a Naval  Aviation 
Maintenance  Training  Detachisant  (NAMCRAOEI),  ^ich  is  a technical 
school  that  supports  the  squadron  at  a Naval  Air  Station.  Typically, 
a NAKCRADEI  includes  several  classrooios,  each  with  chalkboards,  slide 
projectors,  and  other  coosaon  classroom  aids.  The  student  is  provided 
with  a variety  o£  books,  pamphlets,  maintenance  instruction  manuals 
(MlHs),  and  other  material  to  support  his  training.  In  recent  years, 
the  courses  have  included  lists  of  instructional  objectives  (also  called 
behavioral  objectives,  or  specific  behavioral  objectives)  so  the 
student  has  a rather  precise  description  of  what  he  has  to  learn  and 
what  tasks  ha  must  learn  to  perform.  Aircraft  consonants  are  also  on 
display  or  available  for  “haiKls-on''  Instruction  in  the  classroom.  An 
aircraft  wing  (costing  perhaps  a million  dollars),  or  a radar  system, 
or  an  ejection  seat  mounted  in  a section  of  an  aircraft,  or  part  of  a 
computer  system  might  be  found  in  a typical  classroom.  Oscilloscopes, 
pressure  gauges,  and  other  test  equipments  are  also  available. 

In  Fiscal  Year  1974,  1300  instructors  at  49  NAHTRADETS  presented 
818  different  courses  in  aircraft  maintenance.  Each  of  the  818  courses 
was  taught  several  times  for  a total  of  9,077  course  presentations. 

There  were  53,750  graduates  of  these  courses  in  FY74. 

Compared  with  conventional  classroom  training,  the  number  of 
students  attending  a course  is  very  low,  usually  5 or  6. 

The  duration  of  a course  at  a NAMTRADET  depends,  naturally,  upon 
the  coo^lexlty  and  depth  of  training.  Maintenance  courses  for  the  A*7 
aircraft,  for  example,  range  from  8 hours  to  480  hours.  In  FY74  the 
average  course  length  was  111  hours. 

The  students  attending  the  courses  have  varied  backgrounds.  Most 
are  high  school  graduates.  However,  one  student  may  be  fresh  from  a 
technical  school  with  approximately  9**17  weeks  of  technical  training 
whereas  another  student  may  have  had  several  years  of  Navy  maintenance 
experience  behind  him. 

Operational  Equipment  and  Maintenance  Training  Simulators 

The  term  operational  equipment  refers  to  real  aircraft  equipment 
or  real  test  or  other  ground-support  equipment.  Vfhen  operational 
equipment  is  used  for  training,  the  equipment  may  be  in  its  normal 
condition  or  it  may  be  modified  some^diat;  e.g.,  the  cover  of  the 
equipment  may  be  removed,  the  equipment  may  be  partially  disassembled, 
equipment  that  is  normally  adjacent  to  the  equipment  under  study  may  be 
removed. 
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A Maintenance  training  simulator  le  a training  device  that  repreeante 
the  coi^>onenta  and  the  functioning  of  operational  equlpnent.  Although 
physical  fidelity  may  not  be  a necessary  requirement,  siaulators  typically 
represent  faithfully  some  or  all  of  the  components  of  the  operational 
equipment.  The  student  interacts  with  the  simulator  to  acquire  the  know- 
ledge and  skills  required  for  his  maintenance  tasks.  Maintenance  training 
simulators  vary  greatly  in  their  eoaq>lexity,  training  capabilities,  and 
costs. 

In  the  past  few  years,  several  RAMTRAI^S  have  Incorporated  mainte- 
nance training  simulators  into  their  courses.  The  simulator  nmst  In  use 
at  NAMTkADCTS  Is  the  EC  H simulator  of  the  Education  Computer  Corporation, 
^Ich  currently  costs  about  $70,000.  The  EC  II  simulator  comes  In  two 
slaes,  a desk-type  version  for  individual  Instruction  and  a large  panel 
(about  four  foot  high  by  ten  foot  wide)  version  for  Instructing  several 
students  simultaneously.  Included  In  the  EC  II  system  are  power  supplies, 
a random  access  projector  system,  and  a computer  that  can  be  progranoed 
by  a cassette  tape.  Each  simulation  model  Incorporates  a display  panel, 
a plctorlal/schematic  model  of  operational  equipment,  emd  means  for 
"hands-on"  interaction  between  the  student  (or  the  instructor)  and  the 
simulator. 

Maintenance  Levels 

The  Navy  maintenance  program  distinguishes  three  levels  of  mainte- 
nance: organizational  level  (0  level),  intermediate  level  (I  levels, 
and  depot  level. 

0- level  maintenance  is  maintenance  performed  at  an  aircraft  squadron 
on  a day-to-day  basis  in  support  of  its  own  operations.  0-level  maintenance 
includes  equipment  inspections,  equipment  servicing,  equipment  handling, 
corrective  and  preventive  maintenance  for  equipment  on  board  the  aircraft. 

At  the  0-level,  defective  components  are  typically  identified, removed, 

and  replaced.  The  expression  "remove  and  replace"  is  often  used  to 
characterize  0-level  maintenance,  but,  as  Indicated, above,  0-level 
oalntenance  Includes  other  functions. 

1- level  maintenance  Is  performed  by  shops  in  support  of  the  aircraft 
squadron.  Typically,  most  I-level  maintenance  is  performed  on  components 
that  have  been  romoved  from  the  aircraft.  I-level  maintenance  Includes 
repair,  test,  modification,  calibration,  and  qualification  testing  of 
components. 
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; D«^t-l«vl  BMtnf  n«nc«  1«  oalntsoane*  for  •ysteos  requiring  ntjor 

overhaul  or  a coiqplote  rebuild  of  parts « assemblies » subassemblies » or 
end  items  that  is  beyond  the  capability  of  I*level  maintenance. 

I Each  maintenance  task  is  designated  es  0 level.  I level,  or  depot 

level,  so  each  worker  knows  the  limits  of  his  task  and  knows  when  to 
pass  on  the  maintenence  task  to  the  next  higher  maintenance  level. 

. Here  is  an  example  of  O^level  and  I^level  maintenanco  tasks.  The 

[ engine  life  recorder  (ELR)  of  the  AV-dA.  Harrier  aircraft  propulsion 

I system  measures  the  used  life  of  the  aircraft  engine.  Sensors  in  the 

> engine  measure  the  heat  and  tijne  and  convert  that  information  to  a count 

I that  is  displayed  on  a digital  counter. 

' At  preflight  and  postflight  inspections,  the  0»level  technician 

teats  the  ELK  for  accuracy  by  measuring  the  count  rate  when  a ^ullt~in 
[ test  switch  is  in  each  of  two  test  positions.  If  the  EUR  fails  the 

I teats,  the  technician  replaces  it  and  sends  the  defective  EUl  to  the 

I**lavel  madLntenance  shop.  The  0**level  technician  also  logs  the  EIA 
^ count  at  prefli^t  and  postfli^t  and  performs  scheduled  checks. 

1 The  I»level  technician  troubleshoots  the  malfunctioning  EUl. 

replaces  the  defective  subassembly,  and  verifies  his  repair  by  performing 
I a functional  test.  If  the  EIA  cannot  be  repaired  by  the  maintenance 

t procedures  approved  for  I-level  maintenance,  the  technician  ships  the 

I defective  EIA  to  the  depot  maintenance  shop.  The  I»lavel  technician  is 

t also  responsible  for  testing  each  new  ELR  that  enters  the  supply  system 

I for  his  squadron. 
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qUESTIOtOims  ONE 


Th«  ala  of  this  study  Is  to  obtain  aacpart  opinion  on  th«  use  of 
alanlators  varsus  tha  usa  of  oparatlonal  equlpmsnt  In  aviation  oalntenanca 
training. 

This  study: 

a)  Is  to  loalntenanca  training  at  NdMTHADEIS, 

b)  iacludas  both  0«*letval  and  l*leval  (but  not  depot  level)  aalntenance 
training,  and 

e)  is  United  to  slnulator  syseans  that  cost  no  asra  than  $200,000. 


This  first  quastlonnaira  consists  of  flva  questions.  QxMStlons  I 
through  *3ra  dlsctission  questions.  In  answering  these  questions,  pre- 
sent any  ideas,  opinions,  facts,  anecdotes,  guesses,  or  hunches  that 
3rou  believe  are  relevant.  If  you  can  support  a stateoant  with  facts  or 
logic,  do  so;  if  not,  include  the  statement  anyhow.'  Consider  yourself 
to  be  talking  "off  the  record";  no  statement  you  make  will  be  attributed 
to  you  by  name.  In  short,  in  answering  Questions  I through  17,  err  in 
the  direction  of  giving  too  much  rather  than  too  little  information. 

We  can  always  ignore  what  we  consider  sxiperfluous,  but  your  opinion  will 
not  oe  heard  if  you  fail  to  state  it. 

We  are  interested  in  your  personal  opinions,  not  the  opinions  of 
your  supervisor  or  yotn;  colleagtMS  or  anyone  else.  So  you  will  not 
influence  or  be  influenced  by  others,  pl^se  do  not  discuss  this  or 
•subsegiient  queatiowweires  with  anyooew 

In  responding  to  Questions  I through  IV  indicate  if  (and  where) 
you  feel  distinctions  between  0 and  I levels  maintenance  training  are 
required. 

To  lessen  your  task  on  this  first  questionnaire,  dbn*t  concern 
yourself  with  writing  well.  All  that  is  necessary  is  that  we  are  able 
to  discern  your  meaning.' 

You  nay  write  or  type  yr.<ur  axuswecs  to  Questions  I through  IV.'* 
Bowecar,  if  you  prefer  to  dictate  your  answers,  please  do  so.  Send  us 
a rjaaarta  of  youT  remarks.'  If  you  want  the  tape  (or  a blank  tape) 
returned,  please  indicate  this  on  the  tape.'  Also,  give  us  your  nasM  at 
the  beglmlng  of  the  tape.' 


Question  V is  a specific  question  with  instructions  for  answering. 

In  responding  to  questions  I throu^  V,  feel  free  to  add  additional 
sheets  if  necessary  and  oaks  any  conments  about  the  questions  (general 
or  specific) • You  are  also  encouraged  to  cosnent  on  any  questions  you 
feel  we  should  have  asked  but  have  not. 

There  are  many  terms  used  in  discussion  of  maintenance  training 
and  simnlators.'  The  following  is  a list  of  a few  of  the  terms  that 
are  used.  You  nay  wish  to  consider  some  of  these  items  when  responding 
to  the  questions. 

Roles  of  NAMTRADEr  training  experts  (instructors),  aircraft  nanu- 
factinmr,  simulator  manufacturer  and  others. 


Attitudes 
Recurring  costs 
Fidelity 
Versatility 
'*Bands*On'' 

Reliability 

Maintenance  of  training  equipment 
Updating 

Life  span  of  curriculum 

Procurement 

Accessibility 

Fault  insertion 

Safety 

Power  Requirements 
Educational  strategies 
Individualized  training 
Transfer  of  training 


Theory  of  instruction 
Prograsming 
Task/Sklll  analysis 
Training  Objectives  (Specific 
Behavioral  Objectives) 

Class  size 
Instructor  training 
Administration  of  training 
Evaluation 
On  the  Job  training 
Squadron/shop  performance 
Future  state  of  the  art 
S taxkdardlzatlon 
Uniformity  of  training 
Feedback 

Measures  of  student  behavior 


NAME 


i 

I 


!•  0«naral  Cooparlsons* 

If  70a  were  serving  as  an  advisor  to  Naval  adialnlstrators  and  were 
asked  to  list  the  advantagea  and  disadvantages  of  simulators  and  the 
advantages  and  disadvantages  of  operational  equlpoiant  for  aviation 
maintenance  training,  how  would  you  respond? 


r 


I-  i 

i t 

( ! 


} 


! 


I 

I i 
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II.  Factors  In  Daelslon  Making* 

This  quastloa  deals  with  how  tha  daelslon  should  ba  nstda  to  usa 
oparatlonal  aqulpnent  or  sionilators  for  nailntananea  training*  What 
Infbtnatlon  should  tha  daclslon«aaker  hava  available  to  hia?  What 
sources  of  Infonaatlon  axe  likely  to  ba  nost  valuable?  What  factors 
aira  most  critical  in  the  decision  process?  Which  are  least  critical? 
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mV  Specification  for  Simulator  System. 

An  administrator  dacldas  a simulator  rather  than  operational 
equipment  should  be  used  for  maintenance  training  in  a particular 
NAMXRAOEI  course.  What  Items  should  be  included  In  the  contract  for 
the  simulator  system?  What  personnel  should  contribute  to  the 
specification  for  the  simulator  system? 


L3 


Introduction  of  Slnmlators.' 


W. 


t 

When  slsnlaters  axa  Introdxieed  (or  in  us«)  at  NAMTHADEIS  there 
are  sometiino  obatacles  \dilch  hinder  ^alr  acceptance  and  use,  What  are 
sons  probleea  vhlch  might  hinder  acceptance  and/or  use?  What  actions 
can  be  taken  at  your  level  to  reduce  these  problems?  What  actions  can 
be  taken  by  higher  authority? 


V.  The  following  question  asks  you  to  rate  the  feasibility,  the 
training  effectiveness,  and  the  coat  of  simulation  for  training  for 
each  of  ten  aircraft  systems.  Each  of  the  ten  systems,  which  are 
briefly  described  on  the  following  page.  Includes  a variety  of  equip 
msnt.  You  may  have  different  judgements  for  different  equipments 
within  one  system.  If  so.  your  rating  should  apply  to  most  of  the 


equlpinent  within  a system. 


Two  answer  sheets  are  provided.  One,  as  marked,  applies  to  your 
ratings  for  organizational-level  maintenance  training;  the  other  applies 
to  Intermediate-level  maintenance  training. 


The  first  column  of  each  answer  sheet  Is  labelled  “totally  Ignorant 
of  this  system"  • Check  this  column  only  If  you  have  no  knowledge  at  all 
of  the  system.  You  do  not  have  to  be  an  expert  In  the  particular  system 
to  provide  ratings.  Your  general  knowledge  of  a system  Is  sufficient 
for  you  to  participate. 


Here  Is  an  example  of  ratings  for  a system: 


These  ratings  indicate  that  simulation  Is  definitely  feasible  (for  nust 
or  all  of  the  system),  that  simulation  Is  aa  effective  as  operational  > 

equipment  for  training,  and  that  simulation  Is  much  less  expensive  than 
operational  equipment. 


BRIEF  DESCRIPrZORS  OF  AIRCStAFT  MUDHEICWCE  SYSTEMS 


Sttuetur*.  Hydnwllcg  and  Flight  Controls  Syatana  - Includes  for 
exaaple:  slrfraoie  structure  components,  corrosion  control,  hydraulic 
power  supply  and  distrlhotidn  systems,  slitting  and  launching  gear, 
wing  and  fin  I fold  systems,  primary  and  secondary  fli^t  control 
systaiBS,  empennage. 

Propulsion  Systems  * includes  for  example  t engine  maintenance,  quick 
engine  change,  assembly  and  buildup,  rmaoval/installation,  aircraft 
fuel  storage  and  transfer,  fuel  system  operation  and  maintenance. 

Electrical  and  Instananent  Systems  • includes  for  example:  drive  generator, 
electrical  power  supply  axid  distribution,  instrument  indicating  systems, 
lighting,  and  electrical  circuits  for  following  systems;  •wg'tna  and  re* 
lated  system,  hydrattlie,  environmental,  life  support,  azmament, 

Bavironmental  Control  and  Egress  Systems  * includes  for  example:  cabin 
air  conditioning,  cabin  air  presstsizatlon,  crew  egress  component/sub* 
systems,  oxygen  system, 

ArmamentATeapons  Delivery  System  * Includes  for  example:  guns,  mounts, 
weapon  direction  equipment,  launcher,  pods,  bomb  racks  and  other 
mechanical  or  electro*mechanlcal  aqiilpment  for  weapons  delivery  function, 

COBSBunications  and  Navigation  Svstwns  * Includes  for  example;  intercom, 
radio  8ystem(s),  data  link,  radar,  radio,  direction  finding  set,  doppler 
compass. 

Automatic  Flirfit  Controls  * Includes  for  exaiaple;  automatic  pilot, 
fll^t  displays,  mechaniMl  and  electrical  parts  for  signal  transmission 
and  application  of  power,  reference  sensors,  air  data  conq)uter, 

Reconnaiasance  Equipment  * equipment  necessary  to  reconnaissance  mission. 
Includes  for  example:  photographic  and  electronics,  infared  and  other 
sensors,  search  receivers,  recorders,  warning  devices,  magazines  and 
data  11^, 

Anti-Submarine  Warfare  - equipment  peculiar  to  the  antisubmarine  warfare, 
mission.  Includes  for  exanqple:  acoustic  and  xu>nacoustle  sensor  systems, 
computer,  displays. 


ORGANIZATIONAL  LEVEL 


(Sdfex  to  earlier  descriptions  for  distinctions  bet»een 
0-level  and  I-level  maintenance) 


Coopered  to  operational  equip* 
ment  simulators  are: 
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Structure,  Bydranllcs, 
& Fll^t  Controls 


Propolsien 


Slactrlcal  & Instmaant 


Environmental/Egross 


ArwaiaantAyeapons 

Deliver7 


<ni«  Tv< 

Savleatlon 


Antonatle  Fli^t 
Controls 


Reconnaissance 


dntlsulnaBrlne 

Warfare 


INTERMEDIATE  LEVEL 


(^£ar  to  earlier  deeeriptlone  £or  distinctions  between 
0-level  and  I-level  siaintenanca) 


Structure,  Hydraulics, 

& Tll^t  Controls 

Propulsion 

j Electrical  6 Instrunent 

! 

; Esvlrociaental/Egress 

t 

1 

AzsMBient  A^eapons 

Delivery 

Revlgatlon 

Aotooatlc  Fli^t 

Controls 

1 Reconnalssanee 

1 AatlsnbBarina 

1 WCrfaro 

1 

■ 

Cospared  to  operational  equip 
Blent  sisulators  are: 


Equally  expensive 
, Leae  expenaive 
Much  leea  expensive 
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Questionnaire  Two  and  the  material 
provided  with  Questionnaire  Two 
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The  sections  on  the  Delphi  Technique  (given  in  the 
text  of  this  report)  and  on  the  background  material 
(given  in  Appendix  B)  are  omitted  from  this  appendix. 
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Dear 


Thank  you  for  completing  Questionnaire  One  in  our  Delphi  Study  of 
simulators  and  operational  equipment  in  aviation  maintenance  training. 

We  encountered  a few  minor  problems  with  Questionnaire  One  because 
some  of  the  experts  failed  to  follow  directions.  For  example,  when 
completing  the  table  of  Questionnaire  One,  one  expert  decided  that 
the  system  "Structures,  Hydraulics,  & Flight  Controls"  was  too  broad 
so  he  split  it  into  two  systems  and  then  responded.  Since  his 
responses  could  not  be  integrated  with  the  responses  of  the  other 
experts,  his  responses  to  this  item  had  to  be  discarded.  Another 
problem  was  that  a few  of  the  experts  failed  to  complete  all  items. 

In  completing  Questionnaire  Two,  which  is  enclosed,  please  follow 
directions  carefully.  If  you  have  any  comments  about  any  of  the 
items  of  Questionnaire  Two,  first  respond  to  the  item,  then  give 
your  comment,  either  in  the  margin  or  on  the  back  of  the  sheet. 

Please  complete  Questionnaire  Two  promptly,  within  a couple  of 
days  if  possible.  Your  cooperation  and  quick  response  will  be 
greatly  appreciated. 


Sincerely, 


ROBERT  M.  HERRICK 


The  blue  sheets  describing  the  background  material  and  the 
Delphi  Technique,  which  were  included  with  Questionnaire  One, 
are  included  again. 
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Introduction  to  Questionnaire  Two 


The  aim  of  this  study  is  to  obtain  expert  opinion  on  the  use  of  sim- 
ulators versus  operational  equipment  in  aviation  maintenance  training. 

This  study: 

a)  is  limited  to  maintenance  training  at  NAMTRADETS, 

b)  includes  both  0-level  and  I-level  (but  not  depot  level) 
maintenance  training,  and 

f ’ c)  is  limited  to  simulation  systems  that  cost  no  more  than 

‘ $200,000. 


questionnaire  Two  consists  of  three  sections.  In  writing  Section  I 
we  imagined  a naval  administrator  confronted  with  a choice  between  simu- 
lators and  operational  equipment  for  a NAMTRADET  course.  We  ask,  in  this 
section,  what  information  is  needed  to  aid  the  administrator,  and  how 
difficult  is  it  to  obtain  such  information.  Also,  in  Section  I,  we  ask 
about  possible  sources  of  information,  both  personnel  and  relevant 
documents. 

In  Section  II  we  assume  that  the  administrator  has  decided  to  use  a 
simulator  rather  than  operational  equipment.  And  we  ask:  What  issues 
should  the  administrator  consider  to  Implement  his  decision?  And,  how 
frequently  have  these  issues  been  considered  in  the  past? 

In  Sections  I and  II  no  distinctions  are  made  between  0-level  and 
I-level  maintenance  training.  In  giving  your  opinions  in  these  two 
sections  consider  each  item  to  apply  to  both  0-level  and  I-level  training. 

Section  III  presents  the  results  of  Question  Y of  Questionnaire  One 
and  gives  you  the  opportunity  to  retain  or  revise  your  opinions. 
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Specific  Instructions  for  Completing  Table  I 


I 


Table  I concerns  gathering  and  evaluating  information  to  help  decide 
between  simulators  and  operational  equipment. 

An  illustration  will  indicate  the  procedure  you  are  to  follow  in 
completing  Table  I.  Reproduced  below  are  two  items  of  Table  I. 


Item  of  Information 

Need  for 
information 

Effort  required  toj 
obtain  information] 

H 

H 

m 

1 

.-1 

cJ 

0 

CO 

• 

tn 

23.  The  funds  available  for  the  project 

X 

1 

■ 

D 

24.  The  cost  to  buy  each  device 

m 

n 

11 

III 

H 

_J 

1 

a 

LI 

[The  column  headings  (e.g.,  1.  CRUCIAL— the  information  must  be  obtained) 
have  been  omitted  here.  Refer  to  the  first  page  of  Table  I for  the 
complete  headings.] 


In  the  above  illustration  the  expert  decided  that  item  23.  "The  | 

funds  available  for  the  project"  is  information  that  must  be  obtained.  He  j 

therefore  placed  an  X in  the  box  1.  CRUCIAL— the  information  must  be  obtained. 

For  item  23.,  he  also  decided  that  it  would  require  slight  effort  to  obtain  ' 

such  information.  Therefore,  he  marked  X under  2.,  requires  SLIGHT  EFFORT 
to  obtain  information. 

i 

In  each  row  of  Table  I you  are  to  follow  the  same  procedure,  namely,  . ' ^ 

mark  an  X for  one  of  the  five  alternatives  under  the  category  "Need  for 
information."  and  mark  an  X for  one  of  the  five  alternatives  under  the 
category  "Effort  required  to  obtain  information." 

In  many  of  the  items  we  use  the  word  device  in  a special  way.  By 
device  we  mean  either  a simulator  or  operational  equipment.  Thus,  in  item 
24.  above,  "The  cost  to  buy  each  device"  means  the  cost  to  buy  a simulator 
or  the  cost  to  buy  operational  equipment.  Instead  of  listing  two  items 
separately,  one  for  simulators  and  one  for  operational  equipment,  we  list 
only  one  item.  However,  we  split  the  section  for  your  responses  into  two 
rows,  one  marked  "S"  for  simulators  and  one  marked  "0"  for  operational 
equipment.  Fill  in  each  row  as  described  above.  In  the  example  given 
above,  in  the  category  "Need  for  information,"  the  expert  marked  1.  CRUCIAL 
for  both  simulators  (the  row  marked  "S")  and  for  operational  equipment 
(the  row  marked  "0").  In  the  other  category,  "Effort  required  to  obtain 
information,"  the  expert  marked  4.  MAJOR  EFFORT  for  simulators  and  2.  i 

SLIGHT  EFFORT  for  operational  equipment.  ; 

Comments  about  items  may  be  written  in  the  margins  or  on  the  back 
of  the  pages. 

In  the  blank  rows  at  the  end  of  Table  I you  may  add  new  items. 
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Section  I.  Gathering  Information  to  Help  the  Decision  Maker  Decide  between 
Simulators  and  Operational  Equipment. 

Table  I.  Information  Required  for  Decision  Making 


1 

Item  of  Information 

Need  for 
information 

Effort  required  to 
obtain  Information 

1.  CRUCIAL— the  Information  must  be  obtained 

2.  EXTREMELY  USEFUL— the  information  should  be  obtained 

3.  USEFUL— the  Information  probably  should  be  obtained 

4.  OPTIONAL— the  Information  might  be  obtained 

5.  TRIVIAL— the  Information  need  not  be  obtained 

I.  requires  LITTLE  EFFORT  to  obtain  information 

2.  requires  SLIGHT  EFFORT  to  obtain  Information 

3.  requires  MODERATE  EFFORT  to  obtain  Information 

4.  requires  MAJOR  EFFORT  to  obtain  Information 

5.  requires  EXTREME  EFFORT  to  obtain  Information 

CDURSE  CONTENT  CONSIDERATIONS 

1.  A detailed  description  of  each 
task  the  student  must  perform 

2.  The  time  It  should  take  the  student 
to  perform  each  task 

3.  The  tools,  test  equipment,  and  documenta- 
tion available  to  the  student  during 
training  and  testing 

4.  The  criteria  used  to  evaluate  the 
student's  performance  f r each  task 

5.  The  training  objectives  of  the  course 

6.  The  amount  of  supervision  provided  on 
the  Job 

1 


1 


1 
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Table  I (continued) 


Need  for 
information 


Effort  required  to 
obtain  Information 


Item  of  Information 


7.  A description  of  the  physical  environment 
on  the  job  (e.g.,  heat,  noise) 


8.  The  physical  fidelity  (e.g.,  size,  weight) 
required  to  meet  the  course  objectives 


9.  The  availability  of  other  courses  which 
might  meet  the  objectives  of  the  planned 
course 


10.  The  availability  of  Instructional  aids  (e.g., 
lesson  plans,  movies) 


11.  The  amount  of  feedback  to  the  Instructor 
about  student  performance 


12.  The  amount  of  feedback  to  the  student  about 
his  performance 


13.  Analysis  of  frequent  failures  and  repairs  of 
the  weapon  system 


14.  The  tools,  test  equipment,  and  documentation 
available  to  the  student  to  perform 
each  task  on  the  job 


15.  List  of  malfunctions  (faults)  that  the 
maintenance  man  must  recognize  and 
correct  on  the  job 


16.  Lists  of  faults  that  can  and  cannot  be 
Inserted  Into  the  device 


7.  The  need  to  demonstrate  multiple  faults 
for  a single  failure 


18.  The  need  to  Include  Infrequently  encount- 
ered maintenance  tasks 


19.  The  requirement  for  "hands-on" 
experience 


. The  amount  and  type  of  additional  training 
required  on  the  job 
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Table  I (continued) 


Need  for 
Information 


•ffort  required  to 
Jbtain  information 


Item  of  Information 


21.  The  need  for  self-paced  learning 


22.  The  need  for  alternative  routes 
(i.e.,  branching)  for  learning 


ECONOMIC  CONSIDERATIONS 


23.  The  funds  available  for  the  project 


24.  The  cost  to  buy  each  device 


25.  The  cost  for  spare  parts  for  each  device  IS 


26.  The  cost  to  transport  and  install 
each  device 


27.  The  cost  to  modify  NAMTRADET 
facilities  to  use  device 


28.  Power  and  related  costs  to 
operate  each  device 


29.  The  cost  for  the  routine  maintenance  of 
each  device 


30.  The  cost  to  repair  each  device 
if  it  malfunctions 


31.  The  cost  to  update  each  device  as 
changes  occur  in  the  weapon 
system 


32.  The  cost  and  number  of  instructors 
and  support  personnel  with 
each  device 


LIFE  CYCLE  CONSIDERATIONS 


33.  Political  preferences  for  different 
devices 
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Table  I (continued) 


Item  of  Infonnation 


34.  The  expected  life  of  the  weapon  system 


35.  For  a new  weapon  system,  the  delivery 
dates  for  fleet  introduction 


36.  The  number  of  maintenance  personnel 

required  to  support  the  weapon 
system ^ 

37.  The  time  available  to  train  the 
personnel  required  to  support  the 
weapon  system 


38.  The  length  of  time  required  to  procure, 
deliver,  and  install  the  device 


39.  The  availability  of  information  needed 
to  develop  a training  course 


40.  The  anticipated  frequency  and  complexity 
of  modifications  of  the  weapon 
system 


41.  The  anticipated  capabilities  of 
simulators  in  the  next  few  years 


REPAIR  CONSIDERATIONS 


42.  Estimates  of  frequency  of  repair 
for  each  device 


43.  Estimates  of  average  time  to  repair 
each  device 


44.  Requirements  for  special  tools  and 
test  equipment  to  repair  each 
device 


45.  Requirenents  for  specially  trained 
people  to  repair  each  device 


Table  I (continued) 
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46. 

47. 

48. 

49. 

50. 

51. 

52. 

53. 

54. 

55. 

56. 

57. 


Item  of  Information 


Requirements  for  special  documentation 
to  repair  each  device 

Availability  of  spare  parts  if 
device  requires  repair 

Extent  of  factory  support  required 
to  repair  device 

The  accessibility  of  components  of 
device  requiring  repair 


PHYSICAL  CONSIDERATIONS 

The  dimensions  and  weight  of  the 
device 

The  electrical  or  other  power  require 
ments  for  each  device 

The  problems  associated  with  moving 
and  storing  each  device 

The  durability  of  each  device 
under  frequent  use 

The  number  of  sites  where  each  device 
will  be  used 

The  number  of  students  who  could  use 
each  device  at  the  same  time 

The  number  of  students  to  be  trained 
with  the  device  in  one  year 

The  duration  of  the  course 


58.  The  percentage  of  the  class  time 
the  device  will  be  used 
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Table  I (continued) 


Need  for 
Information 


Effort  required  to 
obtain  information 


Item  of  Information 


59.  The  class  time  that  might  be  saved 
with  each  device 


60.  The  scheduling  flexibility  of  the 
course  (i.e.,  what  will  happen 
if  the  device  is  inoperative?) 


CONSIDERATIONS  ABOUT  INSTRUCTORS 
AND  STUDENTS 


61..  The  extent  of  instructor's  experience 
working  with  maintenance  of  the 
system 


62.  The  extent  of  the  instructor's  experience 
teaching  the  course 


63.  The  attitudes  of  instructors  about 
each  device 


64.  The  amount  of  training  of  instructors 
required  to  use  each  device 


65.  The  educational  background  of 
potential  students 


66.  The  maintenance  experience  of  potential 
students 


67.  The  attitudes  of  students  about  each 
device 


ZD 

H" 

OS 

Ui 

o 

OS 

QZ 

OS 

LU 

H- 

o 

o 

o 

U. 

OS 

lu 

u. 

u. 

LU 

o 

LU 

>• 

u. 

u. 

LU 

Ui 

LU 

LU 

LU 

LU 

LU 

< 

— t 

»— 

UJ 

^ • 

< 

z 

-4 

as 

< 

UI 

h- 

< 

£ 

LU 

o 

X 

OS 

LU 

o 

> 

h- 

& 

LU 

O 

OS 

H' 

LU 

O 

QC 

X 

i/i 

a. 

as 

s 

X 

UI 

ZD 

o 

h- 

£ 

LU 

• 

• 

• 

CNJ 

cn 

un 

cvi 

Specific  Instructions  for  Completing  Table  II 


Table  II  lists  possible  sources  to  be  consulted  when  gathering 
information  to  help  decide  between  simulators  and  operational  equipment. 

An  illustration  will  indicate  the  procedure  you  are  to  follow  in 
completing  Table  II.  Reproduced  below  are  two  items  of  Table  II. 


Need 

to  consult  1 

- - 

n 

■51 

El 

■51 

Source  of 

Information 

1 

■ 

■ 

■ 

■ 

1.  Instructors 

of  the  course 

1 

■ 

■ 

■ 

■ 

2.  Instructors 

teaching  similiar  courses 

1 

1 

1 

1 

1 

[The  column  headings  (e.g.,  1.  MANDATORY— must  be  consulted)  have  been 
omitted  here.  Refer  to  the  first  page  of  TaFTe  II  for  the  complete 
headings.] 

In  the  above  illustration  the  expert  decided  that  item  1.  "Instruct- 
ors of  the  course"  must  be  consulted.  He  therefore  placed  an  X in  the 
box  1.  MANDATORY — must  be  consulted.  In  item  2.  "Instructors  teaching 
similiar  courses,"  the  expert  decided  that  it  would  probably  be  very  use- 
ful to  consult  such  a source.  He  therefore  placed  an  X in  box  2.  PROBABLY 
VERY  USEFUL— should  be  consulted.  In  each  row  of  Table  II,  you  are  to 
follow  the  same  procedure,  namely,  mark  an  X for  one  of  the  five  alter- 
natives under  the  category  "Need  to  consult." 

[Note:  In  Table  II  we  are  concerned  with  the  need  to  consult  each 
listed  source  to  help  decide  between  simulators  and  operational  equip- 
ment. Such  sources  should  also  prove  useful  after  the  decision  has  been 
made  to  buy  a simulator  or  operational  equipmentj 

Comments  about  the  items  may  be  written  in  the  margins  or  on  the 
back  of  the  pages. 

In  the  blank  rows  at  the  end  of  Table  II  you  may  add  other  sources 
of  information. 
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Table  II..  Sources  of  Information  for  Decision  Making 


1. 

T. 

T. 

T. 

T. 

T, 

T. 

T. 

9. 


Source  of  Information 

PERSONNEL 

Instructors  of  the  course 
Instructors  teaching  similiar  courses 
Instructors  who  are  teaching  with  simulators 
Training  analysts 

Personnel  doing  the  maintenance  job  in  the  fleet 

Graduates  of  the  course 

Students  of  the  course 

Aircraft  manufacturer 

Manufacturers  of  aircraft  components 


Table  II  (continued) 


Need  to 
consult 


I 


10.  Navy  supply  specialists 


11.  Navy  procurement  specialists 


12.  Navy  safety  specialists 


13.  Manufacturers  of  simulators 


14.  Navy  school  administrators 


DOCUMENTS 


16.  Maintenance  manuals  for  the  weapon  system 


17.  Flight  manuals  of  the  weapon  system 


18.  Maintenance  Plan  (i.e.,  written  record  of  maintenance  concept 
for  system  and  equipment,  including  maintenance  tasks,  level 
of  repair  analysis,  and  support  and  test  equipment 
requirements) 


19.  Personnel  and  Training  Plan  (i.e.,  written  record  of 
personnel  and  training  required  to  operate  and  support 
the  equipment; 


20.  Technical  Data  Plan(i.e.,  written  description  of  all 
information  aids  necessary  to  operate  and  support 
the  equipment) 


21.  Facilities  Requirements  Plan  (i.e.,  written  record  of 
shipboard,  shore,  operational,  maintenance,  and 
training  facilities) 


Transportation  and  Handling  Plan  (i.e.,  written  record  of 
packaging,  handling,  storage,  and  transportation 


15.  Navy  management 


Source  of  Information 


PROBABLY  VERY  USEFUL 


Table  II  (continued) 


23.  Relevant  government  instructions 


24.  Research  and  evaluation  reports  on  maintenance  simulators 

I 


Specific  Instructions  for  Completing  Table  III 
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In  this  section  we  assume  that  the  administrator  has  considered 
all  the  relevant  information  and  decided  to  use  simulators  rather  than 
operational  equipment.  Table  III  concerns  the  administrator's  role 
in  implementing  his  decision. 

Most  of  the  items  in  Tables  I and  II,  which  were  applicable  to  the 
decision  process,  are  applicable  also  to  the  implementation  process— 
the  process  of  procuring,  installing,  and  using  a simulator  system. 

Rather  than  repeat  the  items  of  Tables  I and  II  in  Table  III  we  have  chosen 
to  list  items  that  sunroarize  much  of  the  material  of  Tables  I and  II.  In 
addition  to  the  summary  items.  Table  III  includes  material  not  considered  in 
Tables  I and  II.  Also,  Table  III  asks  you  to  judge  how  frequently  the  issues 
have  been  considered  in  past  simulation  systems. 

An  illustration  will  indicate  the  procedure  you  are  to  follow  in 
completing  Table  III.  Reproduced  below  is  one  item  of  Table  III. 


Issue  to  be  addressed 

Need  to 
an  isi 

iddress 
LiS 

Past  per 
addressi 

•fom 
nq  e 

lance 
»n  is 

in 

5ue 

CO 

MB 

CNJ 

a 

B 

in 

1 1.  Verify  that  course  objectives  are  met 

1 with  the  simulator  system 

1 

■ 

■ 

■ 

■ 
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■ 

■ 

i 

■1 

I 

I 

1 


I 


[The  column  headings  (e.g.,  1.  CRUCIAL— the  issue  must  be  addressed)  have 
been  omitted  here.  Refer  to  the  first  page  of  TabTeTlI  for  the  com- 
plete headings.] 

In  the  above  illustration  the  expert  decided  that  item  1.  “Verify 
that  course  objectives  are  met  with  the  simulator  system"  is  an  issue 
that  must  be  addressed  in  procuring  simulators.  He  therefore  placed 
an  X in  the  box  1.  CRUCIAL—  the  issue  must  be  addressed.  For  the  same 
item,  he  also  decided  that  the  issue  had  been  infrequently  considered  in 
past  simulation  systems.  Therefore,  he  placed  an  X in  alternative  4. 
issue  INFREQUENTLY  has  been  addressed  in  the  past. 

In  each  row  of  Table  III  you  are  to  follow  the  same  procedure, 
namely,  mark  an  X for  one  of  the  five  alternatives  under  the  category 
Need  to  address  an  issue,"  and  mark  an  x for  one  of  the  five  alternati ves 
under  the  category  "Past  performance  fn  addressing  an  is'sue." 

Comments  about  the  items  may  be  written  in  the  margins  or  on  the 
back  of  the  pages. 

In  the  blank  row  at  the  end  of  Table  III  you  may  add  new  items. 
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Section  II.  The  Implementation  of  a Decision  to  Use  Simulators. 

Table  III.  Issues  to  be  Addressed  in  Procuring  Simulators 


Need  to  address  1 Past  performance  i 
an  issue  I addressing  an  issu 
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Issue  to  be  addressed 
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. Verify  that  course  objectives  are  met  with 
the  simulator  system 


2.  Evaluate  the  training  effectiveness  of  the 
simulator  system  by  testing  student 
performance 


3.  Verify  all  costs  associated  with  the 
simulator  are  within  estimates 


4.  Establish  and  monitor  time  schedules  for 
development  and  installation  of  the 
..  simulator 


5.  Insure  provisions  for  updates  and  changes 


6.  Insure  that  adequate  tools,  documentation, 
spare  parts,  and  personnel  are  available 
for  repair  of  the  simulator 
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Table  III  (continued) 


Need  to  address 
an  Issue 


Past  performance  In 
addressing  an  Issue 


Issue  to  be  addressed 


/. Verify  that  actual  repair  frequency 
and  "downtime"  are  within  estimates 

8.  Insure  the  training  site  Is  adequately 

prepared  for  Installation  of 
; simulator 

9.  Establish  and  monitor  time  schedules  for 

I NAWTRADET  course 

j 10.  Establish  follow  up  procedures  to  monitor 
Instructor  and  student  attitudes  toward 
the  simulator  system 

11.  Insure  adequacy  of  Instructor  training 

for  use  of  simulator  system 

12.  Insure  adequate  human  factors  design 

; requirements  have  been  met 

13.  Evaluate  Instructor  performance 

i TT!  Establish  effective  channels  of  communi- 
I cation  between  Instructors  and  management 

15.  Assess  all  effects  resulting  from  Introduction 

of  simulator  system 

16.  Prevent  overuse  of  the  simulator  as  a 

toy  to  Impress  visiting  dignitaries 


a» 

T9 

iA 

01 

Oi 

iA 

•o 

(/) 

O) 

tA 

Q) 

Oi 

</) 

</) 

iA 

u 

0) 

0) 

0> 

0) 

(A 

u 

u 

U 

— J 

Oi 

•o 

•o 

ZD 

fO 

u 

"O 

Ui 

</) 

oo 

>- 

>• 

0^ 

< 

•J 

UJ 

3 

H- 

> 

>- 

-J 

c 

z 

Ui 

•J 

-J 

< 

H- 

z 

Ui 

z 

< 

•J 

Z 

o 

o 

<c 

-J 

z 

< 

UJ 

O’ 

Ui 

o 

00 

' 3 

00 

UI 

oo 

o 

cr 

u. 

: > 

O' 

< 

a: 

i 

ut 

UI 

o 

u. 

X 

Q. 

-J 

CJ 

o 

UJ 

o 

1— 

< 

' u. 

o 

< 

• 

* 

• 

1 • 

CSi 

CO 

IT) 

CSi 

00 

Instructions  for  Completing  Section  III 


Results.  Question  V of  Questionnaire  One  asked  you  to  rate  the  feasi- 
bility,  the  training  effectiveness,  and  the  cost  of  simulation  for  nine 
aircraft  systems.  The  opinions  of  all  participants  are  summarized  in 
two  tables  given  in  this  section. 

In  these  tables  each  entry  gives  the  percentage  of  experts  selecting 
that  alternative  for  a given  category,  such  as  feasibility.  For  example, 
for  the  Organizational  Level  table  for  Structures,  Hydraulics,  & Flight 
Controls,  under  the  category  feasibility,  59%  of  the  experts  said  simu- 
lation was  Definitely  feasible.  20%  said  simulation  was  Possibly  feasi- 
ble,.... and  5%  said  simulation  was  Definitely  infeasible  (see  table), 
the  percentages  add  to  101%,  rather  than  100%,  because  each  percentage 
was  rounded  off  to  the  nearest  whole  number.  For  the  same  system  and 
maintenance  level  (see  table),  18%  of  the  experts  stated  that  simulators 
were  Much  more  effective  for  training,  46%  stated  that  simulators  were 
Possibly  more  effective.  10%  stated  that  simulators  and  operational 
equipment  were  Equally  effective. .... 

Delphi . As  discussed  earlier,  the  Delphi  Technique  obtains  the  indepen- 
dent  opinions  of  experts,  summarizes  those  opinions,  and  then  asks  each 
expert  to  reconsider  his  opinion  in  light  of  the  opinions  of  all  the 
experts.  The  aim  of  the  technique  is  to  obtain  agreement  among  the  ex- 
perts, if  such  agreement  actually  exists.  Agreement  is  indicated  by  a 
high  percentage  (perhaps  50%  or  more)  of  the  experts  selecting  the  same 
alternative,  such  as  the  59%  selecting  Definitely  feasible. 

The  Delphi  Technique  is  similar  to  a conference  in  which  each  expert 
presents  his  independent  opinion  on  the  topic  and  then,  after  having 
heard  the  opinions  of  other  experts,  he  is  given  the  opportunity  to  change 
his  opinion. 

Illustration  of  Instructions.  In  Question  V of  Questionnaire  One,  in 
two  tables,  you  gave  your  opinions  on  simulators  and  operational  equip- 
ment. Now  you  are  asked  to  reexamine  your  opinions  in  the  light  of  the 
opinions  of  the  group,  and  if  you  desire,  change  your  opinions.  An 
example  will  illustrate  how  to  change  or  retain  your  opinion. 
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Here  is  a portion  of  the  table  for  Organizational  Level  Maintenance: 


Structure,  Hydraulics, 
& Flight  Controls 


Propulsion 


Feasibilit 


Training  Cost 

Effectiveness  Effectiveness 


liMaiBBllB 


^19 


[The  column  headings  (e.g..  Definitely  feasible)  have  been  omitted  here. 

Refer  to  the  complete  table  for  the  headings.] 

In  the  above  table,  the  circles  in  red  indicate  the  judgements  of 
one  expert  in  Questionnaire  One.  (On  the  complete  tables  the  opinions 
you  gave  are  indicated  by  the  red  circles.)  Thus,  for  Structures, 
Hydraulics,  & Flight  Controls,  this  expert  judges  that  simulation  was 
1.  Definitely  feasible,  that  operational  equipment  and  simulators  were 
3.  Equally  effective  for  training,  and  that  simulators  were  2.  More 
expensive  than  operational  equipment. 

In  the  category  Feasibility,  the  expert  examines  the  percentages 
representing  all  the  experts  and  decides  to  retain  his  original  opinion, 

1.  Definitely  feasible.  He  therefore  makes  no  mark  on  the  table. 

He  also  decides  to  retain  his  original  opinion  on  Training  Effectiveness, 
namely,  3.  Equally  effective.  On  Cost  Effectiveness  he  decides  to  change 
his  opinion  (Simulators  are  2.  More  expensive)  to  the  opinion  that  simu- 
lators are  4.  Less  expensive.  He  ind'cates  this  new  opinion  by  placing 
an  X in  the  box  4. Less  expensive. 


For  the  Propulsion  system  the  expert  retains  his  former  opinions  on 
Feasibility  and  Training  Effectiveness,  but  for  Cost  Effectiveness  he 
changes  his  opinion  from  1.  Much  more  expensive  to  2.  More  expensive 
and  indicates  this  change  by  marking  an  X in  the box  2.  More  expensive. 

If  he  cares  to,  the  expert  may  provide  a conment  on  the  back  of  the 
sheet  explaining  why  he  did  or  did  not  change  his  opinion.  Such  comments 
are  particularly  helpful  if  the  expert's  opinion  differs  greatly  from  the 
opinion  of  the  group. 

You  are  under  no  obligation  to  agree  with  the  other  experts.  If  an 
honest  difference  of  opinion  exists  among  the  experts,  we  want  the  re- 
sults of  the  questionnaire  to  reflect  that  difference  of  opinion. 


D-19 


Summary  of  Instructions  to  Complete  Two  Tables 


1.  The  table  entries  give  the  percentage  of  experts  expressing  a given 
opinion  (e.g.,  59X  of  the  experts  said  Definitely  feasible). 

2.  The  red  circles  indicate  the  opinions  you  expressed  on  Questionnaire 
One. 

3.  Examine  the  table  entries  for  each  aircraft  system  and  category  and 
decide  if  you  want  to  retain  or  change  your  opinion. 

If  you  want  to  retain  the  same  opinion  you  gave  on  Questionnaire 
One,  make  no  mark  on  the  table. 

If  you  want  to  change  your  opinion,  put  an  X in  the  appropriate 


4.  If  your  opinion  differs  from  the  general  opinion  of  the  group, 
if  you  can,  please  provide  a comment  on  the  back  of  the  table  to 
support  your  opinion  or  to  refute  the  majority  opinion. 


R 


Brief  Descriptions  of  Aircraft  Maintenance  Systems 


Structure,  Hydraulics  and  Flight  Controls  Systems— includes  for 
example:  airframe  structure  components,  corrosion  control,  hydraulic 
power  supply  and  distribution  systems,  alighting  and  launching  gear, 
wing  and  fin  foil  systems,  primary  and  secondary  flight  control 
systems,  empennage. 


Propulsion  Systems— includes  for  example:  engine  maintenance,  quick 

engine  change,  assembly  and  buildup,  removal/installation,  aircraft 

fuel  storage  and  transfer,  fuel  system  operation  and  maintenance.  I 

1 

Electrical  and  Instrument  Systems— includes  for  example:  drive  generator,  ; 

electrical  power  supply  and  distribution,  instrument  indicating  systems,  j 

lighting,  and  electrical  circuits  for  following  systems:  engine  and  re-  i 

lated  system,  hydraulic,  environmental,  life  support,  armament.  j - 

Environmental  Control  and  Egress  Systems— includes  for  example:  cabin  ! 

air  conditioning,  cabin  air  pressurization,  crew  egress  component/ sub-  I 

I systems,  oxygen  system.  [ 

[ ' ' 

f , Armament/Weapons  Delivery  System — includes  for  example:  guns,  mounts, 

weapon  direction  equipment,  launcher,  pods,  bomb  racks  and  other  | 

mechanical  or  electro-mechanical  equipment  for  weapons  delivery  function.  : 

I Communications  and  Navigation  Systems--includes  for  example:  intercom,  ! 

radio  system(s),  data  link,  radar,  radio,  direction  finding  set,  doppler  i ‘ 

compass. 

Automatic  Flight  Controls--includes  for  example:  automatic  pilot,  ’ 

flight  displays,  mechanical  and  electrical  parts  for  signal  transmission  1 

and  application  of  power,  reference  sensors,  air  data  computer.  • 

I j 

Reconnaissance  Equipment— equipment  necessary  to  reconnaissance  mission.  i 

Includes  for  example:  photographic  and  electronics,  infrared  and  other 
sensors,  search  receivers,  recorders,  warning  devices,  magazines  and  ; 

data  link. 

Anti-Submarine  Warfare— equipment  preculiar  to  the  anti-submarine  warfare 
mission.  Includes  for  example:  acoustic  and  nonacoustic  sensor  systems, 

' computer,  displays. 
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Before  mailing. 

Please  check  to  see  that  you  have  written  your  name  on  the  first  page 
of  each  table  and  that  you  have  included  all  the  tables  of  Questionnaire 
Two. 


Table  I 

(6  pages) 

Table  II 

(3  pages) 

Table  III 

(2  pages) 

Table  for 

Organizational  Level 

(1 

page) 

Table  for 

Intermediate  Level 

(1 

page) 

1 

I 

s 
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APPENDIX  E 


Questionnaire  Three  and  the  material 
provided  with  Questionnaire  Three 


Thank  you  for  completing  Questionnaire  Two  in  our  study  of  simu- 
lators and  operational  equipment  in  aviation  maintenance  training. 

Questionnaire  Three  is  enclosed.  It  is,  you  will  be  happy  to  hear, 
the  last  questionnaire  of  the  study.  The  time  required  to  complete 
Questionnaire  Three  should  be  much  less  than  that  required  for 
either  of  the  two  previous  questionnaires. 

We  would  be  grateful  for  any  comments  about  Questionnaire  Three  or 
earlier  questionnaires,  or  about  this  method  for  obtaining  expert 
opinion.  Write  your  comments  on  a separate  sheet  of  paper.  This 
is  your  last  chance  to  voice  your  thoughts  about  the  problem  of 
simulators  versus  operational  equipment.  You  will  be  sent  a copy 
of  the  final  report  of  this  study  as  soon  as  it  is  completed. 

Once  again  I must  ask  you  to  return  the  questionnaire  promptly, 
within  a few  days  if  possible.  A quick  response  will  save  me 
(and  you)  the  time  spent  on  a reminder  phone  call  and  will  mean 
you  will  have  the  final  report  sooner. 

Once  again,  thanks  for  your  cooperation  in  this  study. 

Sincerely, 


ROBERT  M.  HERRICK 


Questionnaire  Three 


In  Questionnaire  Three,  you  are  to  reexamine  the  opinions  you  expressed 
in  Questionnaire  Two  and  also  give  your  opinion  on  a few  new  items.  In 
completing  this  questionnaire  remember  that  this  study: 

a)  is  limited  to  maintenance  training  at  NAMTRADETS, 

b)  includes  both  0-level  and  I-level  (but  not  depot  level) 
maintenance  training,  and 

c)  is  limited  to  simulation  systems  that  cost  no  more  than 

$200,000. 

Results.  Tables  I,  II,  and  III  show*  for  each  item  and  category  of 
Questionnaire  Two,  the  percentage  of  experts  that  selected  each  alternative. 
For  example,  consider  in  Table  I,  item  1.  “A  detailed  description  of 
each  task  the  student  must  perform."  In  Questionnaire  Two  for  this  item 
(see  Table  I),  under  the  category  Need  for  Information.  73%  of  the  experts 
selected  the  alternative  CRUCIAL,  13%  selected  the  alternative  EXTREMELY 
USEFUL,  ....  and  0%  selected  the  alternative  TRIVIAL. 

For  each  item,  for  each  category,  the  alternative  that  you  selected 
in  Questionnaire  Two  is  indicated  by  a red  circle. 

Instructions.  These  instructions  for  completing  Questionnaire  Three  are 
similar  to  those  you  followed  for  the  0-level  and  I-level  tables  of 
Questionnaire  Two. 

For  each  item  of  Tables  I,  II,  and  III  examine  the  percentage  data  • 
and  decide  if  you  want  to  change  or  retain  your  opinion. 

If  you  waat  to  retain  the  opinion  you  gave  on  Questionnaire  Two, 
make  no  mark  for  th^t  item. 

If  you  want  to  change  your  opinion,  put  an  X in  the  box  for  the  new 
alternative  you  have  chosen. 

If  your  opinion  differs  from  the  general  opinion  of  the  group,  if 
you  can,  please  provide  a comment  on  the  back  of  the  table  to  support  your 
opinion  or  to  refute  the  majority  opinion. 

When  examining  Table  I,  recall  that  a row  labelled  "S"  means  that  the 
device  referred  to  in  the  item  is  a simulator,  and  a row* labelled  "0" 
means  that  the  device  referred  to  in  the  item  is  operational  equipment. 

When  new  items  have  been  added,  they  appear  at  the  end  of  a table. 

To  indicate  your  opinion  for  each  new  item,  mark  an  X under  one  of  the  five 
alternatives  for  each  category. 
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Section  I.  Gathering  Information  to  Help  the  Decision  Maker  Decide  between 
Simulators  and  Operational  Equipment. 

Table  I.  Information  Required  for  Decision  Making 


Item  of  Information 

Need  for 
Information 

Effort  required  to' 
obtain  Information 

1.  CRUCIAL— the  Information  must  be  obtained 

2.  EXTREMELY  USEFUL— the  information  should  be  obtained 

3.  USEFUL— the  information  probably  should  be  obtained 

4.  OPTIONAL— the  information  might  be  obtained 

5.  TRIVIAL— the  Information  need  not  be  obtained 

1.  requires  LITTLE  EFFORT  to  obtain  Information 

2.  requires  SLIGHT  EFFORT  to  obtain  Information 

3.  requires  MODERATE  EFFORT  to  obtain  Information 

4.  requires  MAJOR  EFFORT  to  obtain  Information 

5.  requires  EXTREME  EFFORT  to  obtai,  information 

COURSE  CONTENT  CONSIDERATIONS 

1.  A detailed  description  of  each 
task  the  student  must  perform 

9 

13 

8 

6 

1 

— ^ 1 

10 

8 

25 

42 

15 

Z.  The  time  It  should  take  the  student 
to  perform  each  task 

i 

■ 

37 

8 

8 

■ 

18 

43 

29 

6 

3.  The  tools,  test  equipment,  and  documenta- 
tion available  to  the  student  during 
training  and  testing 

1 

■■ 

1 

2 

1 

1 

19 

35 

35 

6 

6 

4.  The  criteria  used  to  evaluate  the 
student's  performance  for  each  task 

■ 

■■ 

1 

21 

1 

1 

10 

25 

35 

23 

8 

5.  The  training  objectives  of  the  course 

H 

ifiiH 

8 

8 

0 

2 

18 

25 

33 

24 

0 

6.  The  amount  of  supervision  provided  on 
the  Job 

6 

37 

38 

13 

6 

23 

35 

35 

H 

Table  I (continued) 


Need  for 
information 


Effort  required  to 
obtain  information 


oc  >— 

t:  fc  o “ 
a£  0£  U.  I—  o 

O O Li.  O'  Lb. 

Lu  11.  Ijj  o Li. 


Item  of  information 


—I  —I  LU  Ul  Ul  U. 

_J  Lbl  < _J  ^ LlI  Lbl 

<Z— IZ<Lbll—  < Z 

0£UJ 

oa£ii.i-i>i—  ulli  oqc 

3H-L1JI—  I-Ih-I— iQ  |^1_ 

a£X(ioo.aei-i_jQ  <-< 
ULbISOI—  -JLOZ  ZLbl 


p s t s ^ % £ 


csj  I m I 


i'Tfl 


A description  of  the  physical  environment 
on  the  Job  (e.g.,  heat,  noise) 


The  physical  fidelity  (e.g.,  size,  weight) 
required  to  meet  the  course  objectives 


The  availability  of  other  courses  which 
might  meet  the  objectives  of  the  planned 
course 


The  availability  of  instructional  aids  (e.g., 
lesson  plans,  movies) 


The  amount  of  feedback  to  the  instructor 
about  student  performance 


The  amount  of  feedback  to  the  student  about 
his  performance 


Analysis  of  frequent  failures  and  repairs  of 
the  weapon  system 


The  tools,  test  equipment,  and  documentation 
available  to  the  student  to  perform 
each  task  on  the  Job 


List  of  malfunctions  (faults)  that  the 
maintenance  man  must  recognize  and 
correct  on  the  Job 


Lists  of  faults  that  can  and  cannot  be 
inserted  into  the  device 


The  need  to  demonstrate  multiple  faults 
for  a single  failure 


The  need  to  include  infrequently  encount- 
ered maintenance  tasks 


The  requirement  for  "hands-on" 
experience 


The  amount  and  type  of  additional  training 
required  on  the  Job 


38  21  35  2 4 13  37  25  10 


35  33  25  6 2 21  27  44 


25  37  29 


55  29  10  6 


34  36  30 


20  30  28  20  2 12  53  25  6 


63  23  12  2 0 1 13  17  42  25 


29  42  23 


37 

25 

37 

25 

27 

44 

21 

1 

46 

25 

46 

31 

46 

14 

39 

35 

i 

29 

8 

48 

13 

33 

1 

8 

46 

i 

12 

44 

12 

53 

17 

42 

19 

44 

Table  I (continued) 


Item  of  Information 


I (21.  The  need  for  self-paced  learning 


■ 22.  The  need  for  alternative  routes 

] (i.e.,  branching)  for  learning 


j ECONOMIC  CONSIDERATIONS 

I 23.  The  funds  available  for  the  project 

j — — 

I 24.  The  cost  to  buy  each  device 


25.  The  cost  for  spare  parts  for  each  device  S 


26.  The  cost  to  transport  and  install 
each  device 


27.  The  cost  to  modify  NAMTRADET 
facilities  to  use  device 


28.  Power  and  related  costs  to 
operate  each  device 


29.  The  cost  for  the  routine  maintenance  of 
each  device 


30.  The  cost  to  repair  each  device 
if  it  malfunctions 


31.  The  cost  to  update  each  device  as 
changes  occur  in  the  weapon 
^stem 


32.  The  cost  and  number  of  instructors 
and  support  personnel  with 
each  device 


LIFE  CYCLE  CONSIDERATIONS 


33.  Political  preferences  for  different 
devices 


Need  for 
information 


if fort  required  to 
Jbtain  information 


ac  h- 

H-  o a: 

Q£  li-  i—  o 

O u.  Of 

U-  Ul  o u. 

LU  u.  La  I 

Ul  UJ  u.  ' 


CRUCIAL 

£ 

Ul 

X 

UJ 

USEFUL 

o 

h- 

a. 

o 

TRIVIA 

LITTLE 

SLIGHT 

LU 

O 

s 

QC 

O 

EXTRtM 

cJ 

CO 

• 

lA 

CNJ 

• 

CO 

iA 

18 

29 

18 

27 

8 

4 

16 

36 

24 

20 

20 

29 

27 

16 

8 

4 

12 

42 

32 

xO 

87 

9 

4 

0 

0 

27 

38 

18 

9 

9 

s 

85 

12 

4 

0 

0 

8 

20 

35 

22 

1.6 

u 

81 

6 

12 

0 

0 

18 

41 

27 

8 

6 

s 

65 

31 

4 

~ir 

6 

18 

25  ' 

TT 

0 

63 

29 

8 

6 

0 

12 

29 

39 

12 

8 

s 

44 

23 

25 

6 

i 

12 

•^1 

27 

6 

2 

0 

46 

19 

27 

6 

2. 

14 

45 

35 

6 

" o’ 

s 

60 

23 

15 

2 

0 

6 

19 

44 

23 

8 

■) 

60 

25 

15 

0 

0 

6 

21 

54 

15 

4 

s 

42 

27 

19 

8 

4 

13 

38 

37 

8 

4 

0 

48 

25 

19 

4 

4 

12 

38 

40 

8 

2 

s 

60 

23 

15 

' 2 

0 

2 

15 

54 

15 

13 

0 

60 

21 

15 

2 

2 

6 

25 

54 

10 

B 

s 

57 

35 

8 

“o" 

0 

2 

10 

33 

35 

6 

55 

33 

8 

2 

2 

2 

22 

37 

27 

B 

s 

65 

7i 

8 

0 

2 

2? 

6 

71 

19 

6 

4 

0 

2 

8 

48 

27 

s 

71 

21 

6 

0 

2 

0 

31 

49 

6 

14 

6 

71 

23 

4 

0 

2 

4 

29 

49 

3 

10 

1 

s 

18 

20 

24 

12 

24 

14 

10 

47 

8 

B 

0 

14 

18 

27 

16 

24 

20 

10 

43 

16 
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Table  I (continued) 


Item  of  Information 


34.  The  expected  life  of  the  weapon  system 


Need  for 
Information 


O 


46 


4/) 

>• 

UJ 

X 

UJ 

oc 

X 

UJ 

• 

CM 


38 


<n 


15 


O 


o| 


Ifll 


Efl^ort  required  to 
obtain  Information 


g 


Of 

p 


o 

00 

Cvj 


23 


^0 


UJ 

o 

CO 


o 


o 


23 


s 

u. 

UJ 

UJ 

s 

UJ 

oc 

I" 

X 

•Jj 

uo 


10 


35.  For  a new  weapon  system,  the  delivery 
dates  for  fleet  introduction 


54 


33 


37 


P3 


36.  The  number  of  maintenance  personnel 
required  to  support  the  weapon 
system 


52 


29 


13. 


28 


17 


19 


37.  The  time  available  to  train  the 
personnel  required  to  support  the 
weapon  system 


60 


27 


19  K8 


19 


38.  The  length  of  time  required  to  procure, 
deliver,  and  install  the  device 


69 


25 


15 


42 


75 


19 


17 


46 


33 


25 


39.  The  availability  of  Information  needed 
to  develop  a training  course 


57 


37 


10 


41 


29 


16 


40.  The  anticipated  frequency  and  complexity 
of  modifications  of  the  weapon 
system 


35 


42 


19 


43 


27 


41.  The  anticipated  capabilities  of 
simulators  In  the  next  few  years 


35  23  31  8 


8 P5  37  19 


REPAIR  CONSIDERATIONS 


42. 

Estimates  of  frequency  of  repair 

B 

46 

42 

10 

~ 

a 

12 

35 

for  each  device 

nr 

50 

44 

B 

0 

B 

2 

12 

35 

m 

43. 

Estimates  of  average  time  to  repair 

s 

33 

50 

15 

0 

2 

4 

12 

29 

10 

each  device 

0 

40 

46 

12 

0 

2 

4 

13 

25 

B 

44. 

Requirements  for  special  tools  and 

s 

52 

29 

17 

0 

2 

6 

23 

15 

12 

test  equipment  to  repair  each 
device 

6 

52 

31 

15 

0 

2 

8 

35 

13 

2 

45. 

Requirements  for  specially  trained 

s 

54 

40 

B 

a 

2 

10 

19 

19 

B 

people  to  repair  each  device 

0 

48 

37 

10 

0 

6 

15 

17 

19 

i 
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Table  I (continued) 


Item  of  Information 

Need  for 
Information 

Effort  required  to 
obtain  Information 

1 . CRUCIAL 

2.  EXTREMELY  USEFUL 

3.  USEFUL 

4.  OPTIONAL 

5.  TRIVIAL 

1 . LITTLE  EFFORT 

2.  SLIGHT  EFFORT 

3.  MODERATE  EFFORT 

4.  MAJOR  EFFORT 

5.  EXTREME  EFFORT 

46.  Requirements  for  special  documentation 

m 

28 

32 

6 

H 

B 

22 

Bl 

m 

8 

to  repair  each  device 

a 

a 

24 

43 

6 

H 

B 

30 

32 

18 

4 

47.  Availability  of  spare  parts  If 

m 

m 

Bl 

Bl 

il 

H 

SI 

24 

Ea 

ta 

10 

L_ 

device  requires  repair 

m 

ttl 

15 

2 

SI 

Ea 

31 

ra 

a 

48.  Extent  of  factory  support  requirec 

m 

a 

38 

8 

6 

0 

27 

33 

B 

to  repair  device 

a 

m 

Bl 

ta 

2 

a 

26 

a 

la 

49.  The  accessibility  of  components  of 

m 

m 

23 

44 

6 

2 

[T 

37 

37 

12 

B: 

device  requiring  repair 

[!■ 

ta 

Ka 

a 

E£l 

DB 

E9 

aa 

a 

PHYSICAL  CONSIDERATIONS 

50.  The  dimensions  and  weight  of  the 

m 

SI 

23 

40 

12 

6 

B 

B 

T 

2 

device 

111 

25  _ 

29 

33 

10 

Bl 

m 

B 

B 

0 

51.  The  electrical  or  other  power  require- 

m 

19 

29 

27 

8 

B 

35 

B 

B 

2 

ments  for  each  device 

38 

27 

25 

6 

4 

144 

31 

19 

4 

2 

52.  The  problems  associated  with  moving 

m 

13 

25 

38 

15 

B 

a 

39 

a 

6 

2 

and  storing  each  device 

13 

29 

40 

12 

6 

25 

48 

23 

B 

B 

53.  The  durability  of  each  device 

m 

4A 

46 

6 

B 

B 

B 

27 

B 

22 

T2I 

under  frequent  use 

4A 

46 

6 

B 

B 

30 

46 

14 

B 

54.  The  number  of  sites  where  each  device 

S 

37 

29 

27 

B 

B 

42 

B 

6 

will  be  used 

a 

37 

29 

a 

B 

B 

a 

40 

19 

6 

55.  The  number  of  students  who  could  use 

s 

43 

43 

12 

2 

B 

B 

38 

m 

8 

a 

each  device  at  the  same  time 

• • 

(T" 

41 

47 

10 

2 

B 

B 

42 

27 

8 

a 

56.  The  number  of  students  to  be  trained 

s 

42 

44 

6 

B 

B 

43 

B 

18 

with  the  device  In  one  year 

a 

46 

42 

B 

B 

a 

a 

24 

18 

2 

|57.  The  duration  of  the  course  | 

22” 

32 

B 

B 

a 

31 

~r~ 

0 

58.  The  percentage  of  the  class  time 

□ 

a 

42 

a 

2 

L9 

37 

19 

y 

the  device  will  be  used 

a 

27 

42 

m 

2 

a 

37 

17 

■'i 

Table  I (continued) 


Item  of  Information 

Need  for 
Information 

Effort  required  to 
obtain  Information 

1 . CRUCIAL 

2.  EXTREMELY  USEFUL 

3.  USEFUL 

4.  OPTIONAL 

5.  TRIVIAL 

1.  LimE  EFFORT 

2.  SLIGHT  EFFORT 

3.  MODERATE  EFFORT 

4.  MAJOR  EFFORT 

5.  EXTREME  EFFORT 

59.  The  class  time  that  might  be  saved 
with  each  device 

a 

S 

12 

B 

D 

m 

8 

46 

40 

6 

H 

m 

m 

12 

6 

D 

D 

6 

48 

40 

6 

60.  The  scheduling  flexibility  of  the 
course  (I.e.,  what  will  happen 

If  the  device  Is  Inoperative?) 

a 

a 

m 

14 

2 

D 

D 

20 

54 

18 

3 

XT 

53 

31 

14 

2 

B 

B 

22 

57 

1 

- 

CONSIDERATIONS  ABOUT  INSTRUCTORS 
AND  STUDENTS 


61..  The  extent  of*  Instructor's  experience 
working  with  maintenance  of  the 
system 

14 

— 

31 

35 

12 

8 

24 

33 

33 

10 

0 

62.  The  extent  of  the  Instructor's  experience 
teaching  the  course 

16 

18 

47 

12 

8 

31 

35 

29 

1 

■ 

63.  The  attitudes  of  Instructors  about 
each  device 

a 

Bl 

22 

6 

6 

EQ 

18 

43 

16 

6 

a 

£9 

29 

8 

8 

IS 

27 

39 

12 

6 

64.  The  amount  of  training  of  Instructors 
required  to  use  each  device  - 

1 

m 

Qj 

20 

2 

0 

12 

32 

38 

18 

D: 

39 

31 

25 

2 

2 

18 

32 

40 

"lo" 

D 

65.  The  educational  background  of 
potential  students 

29 

31 

22 

14 

■ 

24 

35 

25 

14 

2 

66.  The  maintenance  experience  of  potential 
students 

27 

22 

31 

16 

1 

24 

22 

37 

16 

2 

67.  The  attitudes  of  students  about  each 
device 

s 

27 

16 

31 

16 

10 

8 

27 

39 

16 

10 

6 

22 

14 

37 

18 

10 

10 

24 

43 

12 

10 

New  Items 

The  likelihood  that  the  students  may 
damage  the  device 

S 

6 

The  likelihood  that  the  device  may 
cause  Injury  to  the  student 

s 

0 

The  capability  of  the  device  for 
demonstrating  safety  procedures 

5 

5“ 

The  ability  of  the  device  to  duplicate 
actual  operating  conditions 

S 

) 

EXTREME  EFFORT 


Section  I.  Gathering  Information  to  Help  the  Decision  Maker  Decide  between 
Simulators  and  Operational  Equipment. 


Table  I.  Information  Required  for  Decision  Making 


Need  for 
information 


Effort  required  to' 
obtain  Information 


Item  of  Information 


COURSE  CONTENT  CONSIDERATIONS 


1.  A detailed  description  of  each 
task  the  student  must  perform 


2.  The  time  It  should  take  the  student 
to  perform  each  task 


3,  The  tools,  test  equipment,  and  documenta- 
tion available  to  the  student  during 
training  and  testing 


4.  The  criteria  used  to  evaluate  the 
student's  performance  for  each  task 


5.  The  training  objectives  of  the  course 


6.  The  amount  of  supervision  provided  on 
the  job 
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38  37  8 8|  4 18  43  29  6 


8 

25 

18 

43 

35 

35 

25 

35 

25 

33 

35 

35 

Table  II  (continued) 


Source  of  Information 


10. 

Navy  supply  specialists 

11. 

Navy  procurement  specialists 

12. 

Navy  safety  specialists 

13. 

Manufacturers  of  simulators 

U. 

Navy  school  administrators 

15. 

Navy  management 

DOCUMENTS 

16. 

Maintenance  manuals  for  the  weapon  system 

17. 

Flight  manuals  of  the  weapon  system 

18. 

Maintenance  Plan  (i.e.,  written  record  of  maintenance  concept 
for  system  and  equipment,  including  maintenance  tasks,  level 
of  repair  analysis,  and  support  and  test  equipment 
requirements) 

19. 

Personnel  and  Training  Plan  (i.e.,  written  record  of 
personnel  and  training  required  to  operate  and  support 
the  equipment) 

20. 

Technical  Data  Plan(i.e.,  written  description  of  all 
information  aids  necessary  to  operate  and  support 
the  equipment) 

21. 

Facilities  Requirements  Plan  (i.e.,  written  record  of 
shipboard,  shore,  operational,  maintenance,  and 
training  facilities) 

22. 

Transportation  and  Handling  Plan  (i.e.,  written  record  of 
packaging,  handling,  storage,  and  transportation 
requirements  for  equipment) 

Need  to 
consult 


§ 

CD 

O 

DC 


CSJ 


CO 


a. 

o 


Table  II  (continued) 


Source  of  Information- 

Need  to 
consult 

1 . MANDATORY 

2.  PROBABLY  VERY  USEFUL 

3.  USEFUL 

4.  OPTIONAL 

5.  UNNECESSARY 

23.  Relevant  government  Instructions 

38 

25 

25 

10 

2 

24.  Research  and  evaluation  reports  on  maintenance  simulators 

49 

22 

22 

8 

0 

1 New  Items 

Research  personnel  experienced  with  simulators  used 
for  training 

Companies  with  the  potential- for  manufacturing  simulators 

• 

}r^ 


I 

\ 


NAME 

Section  II.  The  Implementation  of  a Decision  to  Use  Simulators. 

Table  III.  Issues  to  be  Addressed  In  Procuring  Simulators 


(■ 

1 


I 

\ 


t 

f 


r 


I 

i 

I ^ 


i 

I 

\ 


I 


I 

i 

'1 


I 


Issue  to  be  addressed 

Need  to  address 
an  Issue 

Past  performance  In 

addressing  an  Issue 
• 

1.  CRUCIAL— the  Issue  must  be  addressed 

2.  EXTREMELY  USEFUL— the  Issue  should  be  addressed 

3.  USEFUL— the  Issue  probably  should  be  addressed 

4.  OPTIONAL— the  Issue  might  be  addressed 

5.  TRIVIAL— the  Issue  need  not  be  addressed 

1.  Issue  ALMOST  ALWAYS  has  been  addressed  In  the  past 

2.  Issue  FREQUENTLY  has  been  addressed  In  the  past 

3.  Issue  OCCASIONALLY  has  been  addressed  In  the  past 

4.  Issue  INFREQUENTLY  has  been  addressed  In  the  past 

5.  Issue  ALMOST  NEVER  has  been  addressed  In  the  past 

• 

1.  Verify  that  course  objectives  are  met  with 
the  simulator  system 

96 

2 

0 

0 

2 

18 

14 

37 

20 

12 

2.  Evaluate  the  training  effectiveness  of  the 
simulator  system  by  testing  student 
performance 

83 

15 

0 

2 

0 

24 

12 

25 

22 

18 

3.  Verify  all  costs  associated  with  the 
simulator  are  within  estimates 

44 

33 

17 

2 

4 

12 

30 

26 

18 

14 

4.  Establish  and  monitor  time  schedules  for 
development  and  Installation  of  the 
, simulator 

44 

38 

15 

0 

2 

22 

39 

25 

10 

• 

4 

5.  Insure  provisions  for  updates  and  changes 

65 

33 

2 

0 

0 

10 

27 

31 

12 

m 

20 

6.  Insure  that  adequate  tools,  documentation, 
spare  parts,  and  personnel  are  available 
for  reoair  of  the  simulator 

78 

20 

2 

0 

0 

8 

33 

24 

18 

IS 

i 


I 


{ 

I 

f 

f 


i 
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Items 


Table  III  (continued) 


Need  to  address 
an  issue 

Past  performance  ini 
addressing  an  issuel 

Issue  to  be  addressed 

1 . CRUCIAL 

2.  EXTREMELY  USEFUL 

3.  USEFUL 

4.  OPTIONAL 

5.  TRIVIAL 

1.  ALMOST  ALWAYS  addressed 

2.  FREQUENTLY  addressed 

3.  OCCASIONALLY  addressed 

4.  INFREQUENTLY  addressed 

5.  ALMOST  NEVER  addressed’ 

7.  Verify  that  actual  repair  frequency 
and  "downtime"  are  within  estimates 

29 

41 

27 

1 

2 

6 

20 

33 

29 

12 

8.  Insure  the  training  site  is  adequately 
prepared  for  installation  of 
simulator 

67 

29 

H 

1 

1 

25 

25 

35 

8 

8 

9.  Establish  and  monitor  time  schedules  for 
NAMTRADET  course 

31 

33 

27 

1 

B 

23 

33 

29 

6 

8 

10.  Establish  follow  up  procedures  to  monitor 
instructor  and  student  attitudes  toward 
the  simulator  system 

25 

40 

31 

2 

2 

6 

19 

19 

31 

25 

11.  Insure  adequacy  of  instructor  training 
for  use  of  simulator  system 

67 

21 

8 

2 

2 

10 

21 

35 

29 

6 

12.  Insure  adequate  human  factors  design 
requirements  have  been  met 

37 

50 

8 

1 

2 

6 

17 

38 

27 

12 

13.  Evaluate  instructor  performance 

40 

37 

19 

2 

2 

13 

31 

21 

13 

21 

14.  Establish  effective  channels  of  communi- 
cation between  instructors  and  management 

38 

38 

12 

10 

2 

10 

27 

33 

22 

8 

15.  Assess  all  effects  resulting  from  Introduction 
of  simulator  system 

27 

57 

12 

2 

2 

8 

12 

24 

22 

16.  Prevent  overuse  of  the  simulator  as  a 
toy  to  impress  visiting  dignitaries 


Establish  programs  to  teach  instructors  how 


21  19  33  12  15 


BBBI 


to  use  simulators  for  training 


Personnel  Data  Sheet 


In  the  final  report  on  this  study  we  want  to  identify  the  participants 
and  provide  some  information  on  their  backgrounds.  (The  opinions  and  com- 
ments you  provided  will,  of  course,  remain  confidential;  no  name  will 
be  associated  with  any  coirroent.) 

For  inclusion  in  the  final  report  please  provide  the  following 
information. 

1.  In  the  space  below,  type  or  print  your  rank  or  other  title 
(e.g..  Dr.,  Prof.,  ATCS),  your  name,  and  your  address. 


2.  Please  check  one  item  in  the  list  of  institutional  affiliations 
and  one  item  in  the  list  of  jobs. 


Institutional  Affiliation 


Principal  Job 


Navy, 

Marines 

Anry 

Air  Force 
Industry 
University 
Other  (specify) 


NAKTRADET  instructor 

Administrator 

Researcher 

Project  leader 

University  professor 

Consultant 

Other  (specify) 


3.  Briefly  describe  the  extent  of  your  knowledge  or  experience  with 
simulators,  including  the  names  of  the  simulators. 


Before  mailing, 

Please  check  to  see  that  you  have; 

a)  completed  the  new  items  in  the  tables 

b)  enclosed  Tables  I (6  pages),  II  (3  pages),  and  III  (2  pages) 

c)  enclosed  the  Personnel  Data  Sheet. 

1 

I 

I 


f 

I 
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APPENDIX  F 

Statistical  analyses  of  data 

* 1.  Problem  of  consensus 

One  aim  of  the  Delphi  technique  Is  to  determine  the  Items 

I on  which  the  experts  are  In  agreement.  Complete  agreement 

1 I would  occur,  of  course.  If  all  the  experts  selected  the  same 

■ j alternative.  For  example.  In  response  to  an  Item,  If  the 

I j experts  were  to  select  one  of  five  alternatives  (varying  from, 

I I say,  "crucial"  to  "trivial")  and  all  selected,  say,  alternative 

two  ("extremely  useful"),  complete  agreement  would  exist.  On 
; the  other  hand.  If  the  responses  of  the  experts  were  randomly 

; distributed  among  the  five  alternatives,  no  agreement  would 

exist. 

i - 

i ; To  specify  precisely  what  Is  meant  by  consensus  of  opinion 

! some  statistical  definition  of  consensus  must  be  derived.  Some 

{ authors  have  concltided  that  a consensus  of  opinion  existed  If 

I 50%  of  the  experts  selected  the  same  category.  Other  authors 

I have  used  more  sophisticated  measures  to  define  consensus, 

usually  some  measure  of  variability. 

Consideration  of  the  problem  of  defining  consensus  suggests 
that  two  crucial  variables  must  be  considered:  the  number  of 
experts  participating  and  the  number  of  alternatives  available. 

The  essential  question  Is  whether  the  division  of  the  experts' 
responses  among  the  several  alternatives  Is  likely  to  occur  by 
chance.  If  chance  Is  a likely  outcome,  then  consensus  cannot 
be  said  to  exist. 

2.  Probability  considerations  on  consensus 

Evaluation  of  chance  outcomes  may  be  determined  with  the 
aid  of  probability  theory.  The  assignment  of  experts'  respor4ses 
to  alternatives  may  be  considered  analogous  to  the  classical 
probability  scheme  of  randomly  assigning  balls  to  urns.  If  we 
have  five  urns  (representing  five  response  alternatives)  and, 
say,  20  balls  (representing  the  selections  of  20  experts),  how 
would  the  balls  be  randomly  distributed  among  the  urns?  Ten 
computer  simulations  of  the  problem  (20  balls  randomly  distributed 
among  five  mns)  gave  the  results  indicated  in  Table  F-1.  Table 
F-1  Indicates,  for  example,  that  in  the  seventh  simulation,  of 
the  20  balls,  two  .11s  fell  in  Um  No.  1,  one  in  Um  No.  2, 
nine  In  Urn  No.  3,  three  in  Um  No.  4,  and  five  In  Um  No.  5. 

This  simulation  Is  analogous  to  two  experts  selecting  alternative 
one  (e.g.,  "crucial"),  one  expert  selecting  alternative  two 
("extremely  useful"),  nine  experts  selecting  alternative  three 
("useful") , etc. . 


F-1 


Table  F-1 


li 


! 

l 

1 


Number  of  balls  randomly  assigned  to  each  of  five  urns 
assuming  equal  probability  (p  ■ .20)  that  any  of 
the  20  balls  will  fall  In  any  urn 


Simulation 

number 

Um  number 

1 

2 

3 

4 

5 

1 

3 

3 

5 

6 

3 

2 

6 

3 

3 

4 

4 

3 

1 

7 

4 

4 

4 

4 

6 

5 

5 

0 

4 

5 

1 

8 

2 

5 

4 

6 

0 

2 

3 

8 

7 

7 

2 

1 

9 

3 

5 

8 

6 

4 

3 

2 

5 

9 

3 

3 

5 

7 

2 

10 

4 

5 

3 

3 

5 

Continuing  the  analogy  further,  statistical  measures  of  central 
tendency  and  variability  may  be  computed  for  each  row  of  data  In 
Table  F-1.  To  perform  such  computations,  assume  that  the  Urn  Numbers 
represent  scores;  thus,  Um  No.  1 represents  a score  of  value  1, 

Urn  No.  2 represents  a score  of  value  2,  Urn  No.  3 represents  a score 
of  value  3,  etc.  Assume  further  that  a score  stands  for  a class 
interval  that  extends  half  a unit  below  and  above  Its  nominal  value. 
Thus,  a score  of  1 represents  a class  Interval  extending  from  0.5  to 
1.5  (or  1.4999),  a score  of  2 represents  a class  Interval  extending 
from  1.5  to  2.4999,  etc. 


1 

< 


i 

j 

i 

i 


F-2 


To  describe  the  situation  in  terms  of  the  responses  of  experts  to 
a questionnaire  item,  consider  Simulation  No.  7 of  Table  F-1.  In 
this  case,  two  experts  selected  category  1 ("crucial"),  and  each 
received  a score  of  1;  one  expert  selected  category  2 and  received  a 
score  of  2;  nine  experts  selected  category  3 and  each  received  a score 
of  3,  etc..  From  this  frequency  distribution  of  scores,  one  may 
compute  statistical  measures  to  summarize  the  distribution.  Thus,  the 
mean  score  is  (2x1  + 1x2  + 9x3  + 3x4  + 5x5) /20  or  3.40.  The  median  or 
50th  percentile,  namely,  the  point  below  which  fall  10  scores  and  above 
which  fall  10  scores,  occurs  within  the  class  Interval  nominally  called 
score  3,  namely,  between  2.50  and  3.4999.  Since  three  scores  fall 
below  2.5,  and  nine  scores  fall  within  the  class  Interval,  the  median 
is  2.5  + (7/9) (1)  or  3.278.  Thus,  for  this  row  of  data  of  Table  F-1, 
the  mean  is  3.40  and  the  median  is  3.278. 

Following  similar  reasoning,  one  may  compute  measures  of  variabil- 
ity for  each  row  of  data  of  Table  F-1.  For  example,  for  Simulation 
No.  7 of  Table  F-1,  the  25th  percentile  (P25)  is  2.722  and  the  75th 
percentile  (P75)  is  4.500,  so  the  interquartile  range,  namely, 

P75  - P25»  is  1.778.  [The  smallest  interquartile  range  possible  is 
0.50,  and  this  would  occur  if  all  selections  were  the  same  alternative.] 


3.  Statistical  definition  of  consensus 


Following  the  procedure  described  in  the  preceding  section,  simula- 
tions were  performed  with  the  aid  of  a computer.  The  goal  of  the  : 

simulations  was  to  determine  what  interquartile  ranges  would  be  likely  I 

to  result  on  the  basis  of  chance.  Such  interquartile  ranges,  would,  ^ 

of  course,  indicate  a lack  of  consensus.  However,  smaller  inter- 
quartile ranges,  below  those  likely  to  occur  by  random  assignment, 
can  be  considered  examples  of  consensus.  i 

All  simulations  employed  five  response  alternatives  (five  urns) , 
because  the  questionnaire  items  had  five  categories.  In  one  set  of 
1,000  simulations,  the  number  of  "judgments" (balls)  used  in  each  sim- 
ulation was  35,  which  corresponded  approximately  to  the  nun^er  of 
NON-NAMTRADET  experts.  In  another  set  of  1,000  simulations,  the 
number  of  "judgments"  used  in  each  simulation  was  25,  which  corre- 
sponded approximately  to  the  number  of  NAMTRADET  experts.  In  addition, 
in  a set  of  1,000  simulations,  the  number  of  "judgments"  used  in  each 
simulation  was  20,  as  in  Table  F-1.  This  last  set  of  simulations  was 
Included  to  represent  cases  where  some  of  the  NAMTRADET  experts  failed 
to  respond  to  an  item. 


F-3 


The  results  of  the  simulations  are  summarized  in  Figure  F-1. 
Figure  F-1  shows  that,  with  "judgments"  randomly  assigned  to  any  of 
five  alternatives,  the  interquartile  range  typically  falls  between 
about  1.80  and  3.00.  The  cut-off  point,  selected  as  an  index  of 
consensus,  was  1.2999.  That  is,  in  the  present  study,  if  the  re- 
sponses to  a questionnaire  item  yielded  an  interquartile  range  of 
less  than  1.30,  a consensus  of  opinion  was  said  to  exist.  If  the 
item  yielded  an  interquartile  range  of  1.30  or  greater,  it  was 
concluded  that  no  consensus  existed. 

This  index  of  consensus,  namely,  an  interquartile  range  less 
than  1.30,  is  a stringent  measure,  selected  to  exclude  almost  all 
cases  where  the  results  might  occur  by  chance.  In  the  1,000  sim- 
ulations with  35  "judgments"  none  fell  below  1.30;  in  the  1,000 
simulations  with  25  "judgments",  only  three  fell  below  1.30;  in 
the  1,000  simulations  with  20  "judgments",  only  four  fell  below  1.30. 
The  selection  of  such  an  extreme  measure  of  consensus  means,  of 
course,  that  a questionnaire  item  on  which  a consensus  of  opinion 
really  existed  might  be  classified  as  an  item  of  non-consensus . 

4.  Evaluation  of  difference  between  the  two  groups 

For  each  questionnaire  item,  the  difference  between  the  NAMTRADET 
distribution  of  judgments  and  the  NON-NAMTRADET  distribution  of  judg- 
ments was  evaluated  by  the  Kolmogorov-Smirnoff  (K-S)  two-sample  test. 
The  K-S  two-sample  test  "is  sensitive  to  any  kind  of  difference  in 
the  distribution  from  which  the  two  samples  were  drawn — differences 
in  location  (central  tendency),  in  dispersion,  in  skewness,  etc." 
(Siegel,  1956). 

Occasionally,  not  all  experts  responded  to  an  item.  This  minor 
variation  in  the  number  of  responses  was  ignored  in  order  to  simplify 
comparisons  between  the  two  groups  of  experts.  Also  ignored  was  the 
difference  in  the  two  sample  sizes,  since  this  difference  would  have 
only  a small  influence  on  the  outcome.  The  comparison  between  the 
two  distributions  for  a questionnaire  item  was  considered  to  be 
statistically  significant  if  the  difference  between  the  two  cumulative 
percentage  distributions  differed  by  40  percentage  points  or  more  at 
any  response  alternative.  A difference  of  40  percentage  points 
represents  a difference  significant  at  the  .05  level,  two-tailed  test, 
when  each  sample  contains  25  responses.  (With  n^  * n2  ” 30,  at  the 
.05  level,  two-tailed  test,  the  difference  required  is  37  percentage 
points. ) 
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L Summary  of  comments  made  by  experts 

' I ‘ 

! A.  Evaluation  of  and  skepticism  of  comments  of  others 

— "I  would  have  liked  many  times  to  have  said  I didn't  really 
know  - but  I know  that  I know  better  than  most  people  who 
will  be  less  reticent.” 

— "Maybe  the  other  experts  are  wrong  and  Ignorance  Is  being 
pooled . " 

— "These  kats  don't  know  what  they  are  talking  about." 

— "My  best  response  Is  that  you  have  a mixed  bag  of  respond- 
ents, many  of  whom  do  not  know  where  the  cost  savings 
leverage  rests." 

— "I  am  stunned  that  so  many  are  willing  to  state  the  effective- 
ness of  simulators  Is  so  much  better  than  operational  equip- 
ment. Where  Is  the  data?" 

— "Where  do  all  these  guys  get  the  Idea  that  simulation  Is  cheap?" 

— "I  would  question  the  actual  experience  of  some  of  the  responders 
with  regard  to  operational  equipment  designed  for  training." 

B.  Discussion  about  "the  decision"  objective 

— "Many  items  I marked  trivial  are  crucial  In  determining  train- 
ing course  strategies  but  don't  affect  the  simulator  v.  hardware 
decision." 

— "Most  'trivial'  items  are  germane  to  NAMTRAGRU  Headquarters, 
not  Chief  of  Naval  Material." 

— "I  think  other  people  responded  to  general  course  planning, 
not  specific  decisions." 

— "I  found  it  hard  to  keep  the  overall  objective  in  mind." 

— "These  are  Important  questions  to  the  actual  development  of 
maintenance  trainers — but  don't  appear  very  necessary  to 
making  a choice  between  operational  equipment  or  simulators." 

C.  Difficulty  with  interpretation  of  items 

— "Answer  depends  on  interpretation." 

— "Too  specific." 

--"Bad  term."  1 

--"Question  and  answer  contingent  on  interpretation  of  'need'."  * 
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— "Question  somewhat  ambiguous." 

— "I  interpret  this  as  Congress  not  as  school  commander." 

— "I  am  assuming  you  mean  the  evaluation  for  a particular 
phase  and  not  the  complete  course." 

— "It  is  usually  difficult  to  ask  the  right  questions  and 
even  if  you  do,  you  do  not  know  what  criteria  the  person 
who  is  responding  is  using  for  each  item  on  the  questionnaire." 

D.  Discussion  about  the  "effort  to  obtain"  information 

— "Information  is  not  available  when  decision  must  be  made." 

— "Rarely  is  all  this  information  available  in  the  real  world 
at  the  time  the  equipment  is  selected." 

E.  Most  experts  underestimated  difficulty 

— "This  should  be  easy  to  do  but  just  isn't." 

— "I  think  I rated  this  information  harder  to  obtain  because 
I am  not  satisfied  with  anything  but  'hard'  data." 

— "I  think  the  Navy  doesn' t have  a readily  accessible  system 
to  do  this — the  data  are  probably  there  but  hard  to  get." 

— "For  the  most  part  my  ratings  are  higher  on  the  effort  side 
because  of  bitter  experience  at  trying  to  pry  these  things 
loose  from  the  fleet  and  the  primes." 


F.  Experts  overestimated  difficulty 

— "This  may  not  be  documented  but  it's  usually  easy  to  find  out." 

G.  Ideas  for  new  work,  areas  of  consideration 

— "What  is  missing  or  weak  are  questions  addressing  linkage 
between  recommendation  and  adaptation,  i.e.  who  is  decision 
maker?" 

— "I  do  feel  that  training  simulators  could  have  a major 
impact  on  . . . maintenance  training  (at  all  levels)...  but, 
unfortunately  the  manufacturers  of  these  simulators  (and  even 
the  research  community)  have  totally  failed  to  exploit  the 
potential  capabilities  of  these  systems...  I would  suggest 
that  the  Naval  administrators  first  find  out  how  to  more 
imaginatively  use  simulators  and  how  to  stimulate  a new  and 
more  Instructionally  relevant  generation  of  simulators  to 
be  built. . " 
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— "I  have  always  believed  that  a few  courses  should  be 
offered  in  the  design  of  simulators...  the  need  to 
train  engineers  and  others  to  design  simulators  is  a 
real  need...  I would  suggest  that  your  study  be  the 
beginning  of  a handbook  or  textbook,  to  be  published 
by  a major  technical  publisher...  Such  a test  would  be 
the  guts  of  a course  which  could  be  given  through  an 
extension  division  of  a state  institution.  Such  a 
course  could  run  during  the  summers  and  be  sponsored, 
i.e.,  supported  spiritually  (not  financially),  by  the 
military  training  organizations.  I readily  admit  that 
the  number  of  individuals  reading  the  book  and/or 
taking  the  course  will  always  be  relatively  small. 

Despite  this,  the  need  explored  In  the  context  of  your 
Delphi  study  is  real  and  will  probably  increase  over 
the  next  25  years — our  productive  lifespans." 

— "I  don't  believe  the  Navy  is  taking  advantage  of  the 
technical  skills  available  when  they  (simulators)  are 
designed.  There  is  absolutely  too  much  inferior 
workmanship  in  some  of  the  details.  I have  pointed 
this  out  to  a number  of  people  but  I never  get  any  kind 
of  feedback.  Does  the  Navy  intend  to  let  these  people 
continue  to  put  out  consoles  that  fall  apart  and 
electronic  boards  that  will  only  fit  in  certain  slots?" 

H.  Justification  for  changes/choices  on  earlier  questionnaires 

Changes 

— "My  opinions  on  cost  effectiveness,  which  are  more  liberal 
than  average,  are  based  on  the  data  (limited  data)  which 

are  available  comparing  the  with  MTUs.  These 

data  generally  show  enormous  savings  ratios." 

— "I  have  changed  my  mind.  I talked  with  who 

teaches  and  uses  the  for  the  hydraulics  systems. 

He  has  convinced  me  that  they  are  superior  to  mock-ups, 
for  Instance,  the  ease  of  entering  system  malfunctions, 
upkeep  and  space  requirements." 

— "Marked  wrong  block  first  time." 

Remained  with  first  choice 

— "I  insist,  it  requires  extreme  effort  to  do  this  right." 

— "Provisioning  is  critical." 

— "I  insist,  "hands  on"  experience  is  the  only  way." 


— I do  not  wish  to  change  any  of  the  opinions  expressed 
as  they  are  my  own.  After  15  years  of  working  with 
aircraft...  and  now  working  and  teaching  dally  with 
simulators,  I feel  that  some  areas  of  Instruction 
just  cannot  be  simulated  adequately." 

— "Without  further  amplification  of  the  questions,  I 
find  It  difficult  for  an  expert  to  change  his  mind 
merely  on  the  basis  that  others  checked  a different 
block." 

I.  Skepticism  of  simulator  manufacturers 

— "The  manufacturers  tell  one  story  when  selling  something 
and  another  when  they  have  to  repair  it  or  supply  parts 
for  it." 

J.  Difficulty  with  cost  comparisons 

— (Simulators)  may  not  be  less  expensive  in  the  long  run." 

— I am  somewhat  rusty  when  comparing  the  cost  of  simulators 
and  cost  of  operational  equipment  for  each  particular 
subsystem. " 

— "Development  costs  (for  simulators)  are  high  and  are  in 
addition  to  development  costs  for  the  original  equipment." 

— "Depending  on  the  quality  of  the  training  analyses  and 
the  specifications  derived  from  them,  simulators  could 
be  less  expensive — or  more  cost-effective,  which  you 
probably  should  have  asked  about — then  operational  gear." 

— "I  had  difficulty  with  the  $200,000  limit." 

— "I  think  you  are  mistaken  to  set  a limit  of  200K  per 
device.  Each  requirement  should  be  handled  on  an 
individual  basis." 

K.  Criticism  of  0 and  I Tables 

— "Considering  the  limited  nature  of  0 level  tasks,  simulation 
is  easily  applied  to  all  categories  of  equipment." 

— "In  this  day  and  age  anything  along  the  lines  of  this 
inquiry  is  feasible.  Just  because  something  is  feasible 
does  not  mean  it's  necessary...  each  situation  of  each 
area  of  each  system  must  be  analyzed  and  determined  on 
actual  facts  2md  this  determines  what  method  is  best  to 
present  the  material  to  be  learned." 

— "I  just  don't  feel  comfortable  trying  to  answer  such 
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abstract  questions  about  these  systems,  even  though  I'm 
familiar  with  most  of  the  systems.  Current  state  of  the 
art  probably  makes  simulation  feasible  for  all  such 
systems.  Properly  used  such  simulators  could  be  more 
effective  than  operational  equipment." 
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